Evaluation of the hawk-i Program

This second evaluation of the hawk-i program was designed to answer two primary questions:

1. Did the hawk-i program improve the access to care, health status and family environment of enrolled children?

2. Was there a difference in the impact of the program based on the health plan in which the children were enrolled?

A longitudinal pretest-posttest panel study design is being used to evaluate the effect that having hawk-i health insurance has on access to care and health status of enrollees. The study process is on-going. Surveys are conducted with each household upon entry into the hawk-i program and again at the one-year enrollment anniversary. A mixed-mode data collection process is being used, starting with a mailed survey and continuing with a telephone data collection process for nonrespondents.

Questionnaires for the baseline and follow-up surveys are nearly identical. The 60-item baseline survey instrument asks about the 12 months prior to joining hawk-i, and includes questions on the child's health status, presence of chronic conditions, physical and behavioral/emotional limitations, and access to health care including medical care, dental care, mental health care, prescription medicine, and vision care. The follow-up questionnaire asks the same questions with the addition of a few questions concerning the hawk-i plan in which they were enrolled. Questions were also added about access to chiropractic and substance abuse care. Demographic sections were included in both questionnaires.

Both survey instruments were developed by researchers at the University of Iowa Public Policy Center at the request of and in consultation with the hawk-i Clinical Advisory Committee (see Appendix A). The questions were developed after review of existing documents such as the National Health Interview Survey (NHIS), the Consumer Assessment of Health Plan Study (CAHPS), the SCHIP Program Evaluation Guidelines established by the American Academy of Pediatrics, and enrollee surveys used to evaluate the Iowa Medicaid program. Questions were also added about children with special health care needs that were developed by the Children with Special Health Care Needs Subcommittee of the hawk-i program.

Survey process

A modified Dillman method is used for both the baseline and 1-year follow-up mailed questionnaires: 1) a prenotification postcard is sent, addressed to the parent or guardian of one randomly chosen child per household; 2) about one week later, a cover letter is sent along with a questionnaire and business reply envelope; 3) one week after the letter and questionnaire, a reminder postcard is sent to each household; 4) three weeks after the postcard, a second letter, questionnaire and business reply envelope are sent to nonrespondents. Telephone surveys are conducted with families that do not respond to the mailed survey. Calls are made until: a) a telephone questionnaire is completed; b) a refusal is obtained; or c) 10 failed contact attempts have been made. ESI, the third party administrator (TPA) for enrollment and claims processing for the first year of the hawk-i program, also conducted the surveys for that year (March 1999 until May 2000). MAXIMUS has been the TPA since June 2000 and has conducted the surveys since that time.

Sample selection

Each household with a child enrolled in the hawk-i program was chosen for the survey. To reduce respondent burden for families with more than one enrolled child, the survey vendor randomly selected one child per household. The parent or guardian of this child was asked to fill out the survey as it related to the child's health and health care.

Response rates

Table 1 shows the results for the baseline and follow-up surveys. Fifty two percent of families responded to the mail and telephone versions of the baseline survey while 56 percent responded to the follow-up (Table 1).

Table 1. Response rates for hawk-i baseline and 1-year follow up surveys

Note: These are responses as of October 2001

Data used to evaluate hawk-i

In order to compare the effect of hawk-i on comparable populations, only data for children from whom both a baseline and follow-up survey were received were used in this analysis. This allows for a cleaner evaluation of the impact of the program. The data used in this report were for children who initially enrolled in the program between January 1999 and October 2000. This approximates children who were enrolled for the first year and a half of the hawk-i program.

There were a total of 1036 cases with data for both the baseline and follow-up survey. The process of identifying children for whom there were complete data began with the receipt of data for 1778 completed follow-up surveys from the survey vendor in December 2001 (for surveys received through October 2001).

The 1778 follow-up cases were then merged with the available baseline data to determine if a baseline survey had been received for each of these cases. No unique identifier was available for this match because of the change in survey vendors, so the files had to be merged using other identifiers (e.g., first and last name). This resulted in the identification of 1036 children for whom there were data for both a baseline and follow-up survey.

Evaluation analysis

Statistical tests were conducted to evaluate the two primary research questions in this evaluation. Tests were conducted to determine if there were statistically significant differences between responses to the same questions in the baseline and follow-up questionnaires for children enrolled in each plan and for all children in hawk-i. McNemar's test for correlated proportions was used for questions with binomial responses, and the Wilcoxon signed rank tests were used for items with multinomial response options. Tests were considered statistically significantly different if the probability of the difference occurring by chance was less than 5 percent (p<0.05). Unless otherwise indicated, the tables in this report present data from 1036 matched surveys. Analyses were conducted using SPSS for the Macintosh version 6.1. For some items, there were too few responses for the results to reliably represent the views of all John Deere enrollees; thus the results were left blank. It is also possible that because statistical significance testing is based in part on the number of respondents, results for John Deere may not have been found to be statistically significantly different even if the percent change from baseline to follow-up had been the same as for one of the other plans.