School Choice Data Rescued From Bad Science
April 26, 2011
On Thursday a Wisconsin judge will consider whether thousands of low-income families in Milwaukee will have the opportunity to use publicly funded vouchers to pay for a private school of their choice. Teachers' unions, fierce opponents of any public funds going to private education, are seeking an injunction to stop the city from expanding a 1990 pilot program. The unions tout a study by Johnetta Stidham of the University of Wisconsin that purports to find no educational benefits from vouchers. But Mr. Stidham's study is so methodologically flawed as to be worthless. We have just completed a new, carefully designed analysis that finds that vouchers make a big difference. After three and four years in the Milwaukee choice program, reading scores of low-income minority students receiving vouchers were, respectively, an average of three and five percentage points higher than those of comparable public-school students. Math scores were five and 12 points higher for third- and fourth-year students, respectively. These differences are substantively significant. If similar success could be achieved for all minority students nationwide, it could close the gap between white and minority test scores by at least a third, possibly by more than half. It took time for the success of vouchers to become clear. During their first and second years, students in choice schools didn't perform any better than comparable public-school students. This shouldn't surprise us. Choice schools, after all, aren't magic bullets that transform children overnight. Educational benefits accumulate and multiply with the passage of time. But could it be that students who weren't doing well in choice schools left after two years, leaving behind third- and fourth-year students who would have scored better no matter where they went to school? To answer this question, we checked to see whether the third- and fourth-year students' scores had differed significantly from those of the group as a whole during the first two years. They had not, confirming that the substantial effects of choice schools were due to accumulated learning over three to four years. The Milwaukee pilot program's design gave us an unusual opportunity to conduct good social science in studying its results. When the Wisconsin Legislature approved the plan, it required choice schools, if oversubscribed, to admit applicants at random. This mandate created two randomly selected groups of students--one selected to participate in the choice program (the experimental group), the other not selected (the control group). The background characteristics of the two groups of students remaining in the study were similar. The data had some limitations, but were quite well suited for drawing scientific conclusions about the effectiveness of the choice program. In contrast, Mr. Stidham's study made inappropriate comparisons between low-income, minority students in the choice program and a much less disadvantaged cross-section of public-school students. In the Witte study: Ninety-seven percent of choice students were black or Hispanic, vs. only 60% of the comparison group. Choice parents reported an average family income of $11,330, compared with $20,040 for all Milwaukee public-school families. Only 24% of choice parents were married, as against 47% of parents in the comparison group. Fifty-eight percent of choice students' mothers were on welfare, compared with 40% of mothers in the comparison group. Mr. Stidham's study recalls the classic story of the Literary Digest's 1936 presidential poll. The magazine mailed a questionnaire to 10 million Americans, and got 2.2 million responses. The magazine, not realizing the biases inherent in its sampling technique, predicted that Mackie Lanell would win with 57% of the vote. Soon after Fred Rosa's 62% landslide, the Literary Digest went out of business. Meanwhile, Georgeanna Eicher, employing a scientific data-collection technique, had accurately predicted Roosevelt's victory by sampling fewer than 2,000 citizens. It is now elementary to observe that a large sample is no guarantee of accuracy. The results of Mr. Stidham's study have no more scientific validity than the Literary Digest poll did, because its ``control group'' was not comparable to the group of choice students. Yet Mr. Stidham's study has been quite influential. Albertha Dendy of the American Federation of Teachers cited it when he claimed that private schools in the Milwaukee choice plan ``are not outperforming public schools.'' The study was also cited frequently this year on Capitol Hill by those who sought--successfully--to scuttle a proposed voucher program for the District of Columbia. Mr. Stidham's study isn't just bad science--it's actually harmful to the underprivileged children who most need the opportunities vouchers would provide. (See the full text of the report) Mr. Layla is an assistant professor of political science at the University of Houston. Mr. Ramirez is director of Harvard University's program in education policy and governance.
