Brian Altonen, MPH, MS

In winter of 1994/5 I obtained a perfect score on a PIP I developed for a program devoted to measuring childhood immunization rates for the Medicaid population, for members reaching 2 years of age during the year of study.  It was not so much the content of the report for this project that impressed the reviewers as it was my attention to detail, in particular those details related to the methodology and a number of questions discussion the validity and reliability of your study’s outcomes. 

The terms validity and reliability, along with several other terms in statistics like “Regression to the Means” and Type 1 and Type 2 error are often understood when these reports are generated, and even partially included in the report’s text, but rarely do statisticians interpret their meaning and application to one’s personal methods of producing and evaluating the overall statistical outcomes. 

One of the first things I did for my PIP report is much like I have done for institutional grants that I developed and then submitted for consideration and review.  I developed the information needed to prove my claims or whatever, and made sure to include these materials as part of an appendix.  I made certain the reports were comprehensive and not too lengthy, and that they made heavy use of screen images to get some of my points across to the reviewer.  Most importantly, I disclosed the formulas and methods I used to develop my own statistical tools.  This way, their statistician could check my methodology out for himself/herself.  Since this latter trick in preparing the drafts for my reports to the State or Regional reviewers seemed to always work at the reporting end, I used this same method to develop my research proposal and report for PIPs and QIAs as well.  

In the case of the Childhood Immunization study, the methods used to perform the two standard measures for this study were total clones from the HEDIS and NCQA programs.  For my study, since the interventions taking place involved attention paid to well visits participation, I decided to measure well visit events and related clinical performances for these events to immunization outcomes events.  This seemed to be the most logical thing to do since the intervention activity for this project involved mailing out reminder letters about well visits and their needs, with the unmentioned goal of improving immunization outcomes, not a study of the participation of all well visits.  So, I turned the latter into another measure, and evaluated well visit activity to the chances for successful completion of all immunization requirements by the age of two.

Typically, we would expect that the likelihood of completing your immunization series by the age of two would be directly related to how much you engage in all of the recommended well visits, since these visits are where these immunizations are supposed to take place.  Such was not the case however, as my study showed.  Instead, I found that the chances for completion (or in this case, failures to complete) were directly linked to whether or not certain well visits were skipped.  If you skipped just one or two specific well visits, your chances for completion plummeted signficantly, not because these shots could not be made up later, but rather because missing this visit mean that a certain human behavior and psychology was at play with regard to mother-child-doctor relationships and how the mother viewed the importance of the well visit. 

The most influential well visits, which when missed resulted in a greater likelihood for failure to complete, were the sixth and ninth month visit, not because of how many shots had to be acquired during these visits, but because missing them meant you had an increased likelihood of not participating in subsequent preventive activities, where these missed immunizations could be accounted for before the age of two was reached.   There were after all the 12 month, and 18 month visits, which were typically interpreted as make up visits designed to catch up with missed immunizations and such.  There was also the possibility of a 15 month visit as well, which in the medical records used a form that was a duplicate of the form used to report the fifteenth month visit pretty much.  So if we include the optional 15 month visit into this review, we find the mother had three opportunities to completely immunize her child before the age of two is reached.  C’est la vie.

However, another issue that popped up when immunizations were related to well visits was the misuse or misapplication of immunization products.  It was found in several cases for example that certain physicians like to use a multitude of combination immunization products, thereby meeting the required numbers of shots for one illness, whilst providing one or two additional shots for another.  As a cost-related issue, this demonstrate a problem due to the much higher cost for combination products.  Instead of employed several lost cost products and a few high cost combos, a physician may opt to just use a series of high cost combos, resulting in higher costs to the insurer (in this case Medicaid).  For example, a series that could cost as little as $150.00 back around 10 years ago, would instead cost $250.00 due to these short cuts. 

Another short cut the practitioner took was skipping immunizations by providing combos for 3 or the 5 main immunization-related well visits, or 5 of the 7 required visits, making it so one or two immunizations never achieve their final count of booster shots by the age of two.  In terms of HEDIS and NCQA scoring, this is a non-hit and therefore lowers your score in terms of percent completion.  At the practitioner level, this means that is as little as three major shots, most of what a child needs can be provided, at the risk fo falling short once or twice, which can be made up later in the eyes of the practitioner.  Not sure if this suggests the same level of safety and prevention at the child’s or mother’s level however.

So, by adding well visits to a study of immunization, you answer a series of important questions regarding why immunizations are sometimes not completed by the time a child reaches the age of two years.  The most important question asks ‘why does the mother skip the 6th or 9th month visit?’  Obtaining an answer to this question eliminates about three-fourths of the cases with failed completions of the series. 

The second question to ask is ‘why doesnt the completion and participation in all well visits directly relate to the likelihood of completing immunization series?’ 

his is apparently because those who engage in well visits do not necessarily expect to undergo the needed immunizations, unless they are so informed, perhaps.  The well visit serves as a social activity to many, rather than a preventive activity.  Therefore, it is up to the physician to make sure the right processes are engaged in during a well visit, and that all well visits are attended to by the mother and child.  In particular, the provider has to make sure that two visits are not skipped, namely the 6th and 9th month visits, and that in such cases when these visits are skipped, that an alert be posted in the patients records indicating the need to catch-up, and not just provide the regular immunizations expected for these later visits.

How this related to the scoring of my PIP pertains to the arguments these findings elicited and the subsequent follow-up intervention activities it then led to.  A specific time period in the child’s life was most at risk for failing to complete the series, for which reasons interventions letters were sent for the upcoming 6-month or 9-month visits.  this was sent to all children that fit that age category, whether or not they were included in this study (since this is a 3-year study, these positive results would surface when they turned two, during the third year).  Also included was a letter targeting people about to turn two for that year, informing them of the need to complete the entire immunization series.   After identifying the children who were behind in this series, their PCP was then contacted as well informing him/her of this concern and the need for the immunization series to be completed, by a specific date (the child’s second birthday).

These aftereffects of the original report also resulted in great scores.  However, 90% of the great scores really had to do with the first year and a half of research and intervention activities.  The process itself is typically where people lose points in these QIA and PIP reports.  Even though it is clearly stated in parentheses about what should be included, which is also mentioned as well during any related teleconferences pertaining to this work, people often miss the basics regarding the biostatistical process–those items stated above–validity and reliability, and making mention of “Regression to the Means” should your draft version of this study result in higher scores than the final study performed near the end of the year.