Evaluation of the hawk-i Program

This third evaluation of the hawk-i program was designed to answer two primary questions:

1. Did the hawk-i program improve the access to care, health status and family environment of enrolled children?

2. Was there a difference in the impact of the program based on the health plan in which the children were enrolled?

A longitudinal pretest-posttest panel study design is being used to evaluate the effect that having hawk-i health insurance has on access to care and health status of enrollees. The study process is on-going. Surveys are conducted with each household upon entry into the hawk-i program and again at the one-year enrollment anniversary. A mixed-mode data collection process is being used, starting with a mailed survey and continuing with a telephone data collection process for nonrespondents.

Questionnaires for the baseline and follow-up surveys are nearly identical. The 60-item baseline survey instrument asks about the 12 months prior to joining hawk-i, and includes questions on the child's health status, presence of chronic conditions, physical and behavioral/emotional limitations, and access to health care including medical care, dental care, mental health care, prescription medicine, and vision care. The follow-up questionnaire asks the same questions with the addition of a few questions concerning the hawk-i plan in which they were enrolled. Questions were also added about access to chiropractic and substance abuse care. Demographic sections were included in both questionnaires.

Both survey instruments were developed by researchers at the University of Iowa Public Policy Center at the request of and in consultation with the hawk-i Clinical Advisory Committee (see Appendix A). The questions were developed after review of existing documents such as the National Health Interview Survey (NHIS), the Consumer Assessment of Health Plan Study (CAHPS), the SCHIP Program Evaluation Guidelines established by the American Academy of Pediatrics, and enrollee surveys used to evaluate the Iowa Medicaid program. Questions were also added about children with special health care needs that were developed by the Children with Special Health Care Needs Subcommittee of the hawk-i program.

Survey process

A modified Dillman method is being used for both the baseline and 1-year follow-up mailed questionnaires: 1) a prenotification postcard is sent, addressed to the parent or guardian of one randomly chosen child per household; 2) about one week later, a cover letter is sent along with a questionnaire and business reply envelope; 3) one week after the letter and questionnaire, a reminder postcard is sent to each household; 4) three weeks after the postcard, a second letter, questionnaire and business reply envelope are sent to nonrespondents.

Telephone surveys are conducted with families that do not respond to the mailed survey. Calls are made until: a) a telephone questionnaire is completed; b) a refusal is obtained; or c) 10 failed contact attempts have been made. MAXIMUS, the third party administrator (TPA) for enrollment and claims processing for the hawk-i program, has conducted the data collection for the surveys presented in this report.

Sample selection

Each household with a child enrolled in the hawk-i program was asked to participate in this study. To reduce respondent burden for families with more than one enrolled child, the survey vendor (MAXIMUS) randomly selected one child per household to be the subject of a survey. The parent or guardian of this child was asked to fill out the survey as it related to the child’s health and health care.

Response rates

Table 1 shows the results for the baseline and follow-up surveys. Sixty-six percent of families responded to the mail and telephone versions of the baseline survey, while 50 percent responded to the follow-up (Table 1).

Table 1. Response rates for hawk-i baseline and 1-year follow up surveys

Note: These are responses as of September 2002

Data used to evaluate hawk-i

In order to assess the effect of hawk-i on comparable populations, only data for children from whom both a baseline and follow-up survey were received were used in this analysis. This allows for a cleaner evaluation of the impact of the program. The data used in this report were for children who initially enrolled in the program between July 1, 2000 and October 1, 2001. This approximates children who were enrolled during the second year of operation of the hawk-i program.

There were a total of 2005 cases with data for both the baseline and follow-up survey. The process of identifying children for whom there were complete data began with the receipt of data for 2551 completed follow-up surveys from the survey vendor through September 2002.

The 2551 follow-up cases were then merged with the available baseline data to determine if a baseline survey had been received for each of these cases. This resulted in the identification of 2005 children for whom there were data for both a baseline and follow-up survey

Response Bias

Statistical tests were performed to evaluate whether systematic bias was introduced into the results because only respondents to both surveys were used. Of the 5158 people originally selected for this study, 2005 responded to both the baseline and follow-up questionnaires and were used in this study. Respondents were more likely to be from larger families, have lower incomes, and report their child's race/ethnicity as white. There were no differences between respondents and non-respondents regarding gender, age, English as the primary language, or the health plan in which they were enrolled.

Evaluation analysis

Statistical tests were conducted to analyze the two primary research questions in this evaluation:
a) To determine if there were statistically significant differences between responses in the baseline and follow-up questionnaires, McNemar’s test for correlated proportions was used for questions with binomial responses, and the Wilcoxon signed rank tests were used for items with multinomial response options.
b) To determine if there were statistically significant differences between responses for the different health plans, Chi-square tests were conducted.
Tests were considered statistically significantly different if the probability of the difference occurring by chance was less than 5 percent (p<0.05).
Unless otherwise indicated, the tables in this report present data from 2005 matched surveys. Analyses were conducted using SPSS for the Macintosh version 6.1. For some items, there were too few responses for the results to reliably represent the views of enrollees in a plan; these questions are noted in the table footnotes and the results were left blank