Making Sense of the Lousy Test Results in Louisiana’s Voucher Program – by Frederick Hess

Last week, after four years spent evaluating the Louisiana Scholarship Program (LSP), the School Choice Demonstration Project issued a comprehensive summary of the findings—and the results were pretty grim.

The research team, led by University of Arkansas professor Patrick Wolf, concluded: “Overall, participating in the LSP had a statistically significant negative impact on student English Language Arts (ELA) and math scores across most years of the evaluation, including the fourth year, and across most samples of students studied.” Because Wolf has earned a reputation for both scholarly rigor and fair-mindedness when it comes to evaluating school choice, the negative findings cannot be lightly dismissed as the handiwork of anti-voucher ideologues.

What to make of these results, however, is less straightforward than the findings themselves. Wolf et al. explain that things are more complicated than the overall findings might suggest.

First, test scores are an imperfect gauge of outcomes, as Wolf et al. make clear. For instance, while LSP students posted lower test scores than did their peers, there was no commensurate effect on whether or not participating students wound up enrolling in college. It may be that students are benefiting in ways that don’t show up on the state tests.

Second, it also turned out that the test effects were only half as large when students were assessed using the iLEAP test (which is less aligned to the public school curriculum) than the LEAP (which is more aligned). This means that a substantial chunk of voucher underperformance on state tests might simply reflect that private school students spend a lot less time focusing on preparing for state tests than do students in public schools. What’s striking is that this question of how to make sense of movement in test scores is exactly the same one that arises—and is too often given short-shrift—when it comes to district schools.

Now, given that some choice advocates once insisted that test scores proved that school choice “works,” critics have a point in calling out those advocates who’ve flipped and now insist that it was never about test scores. But I feel comfortable setting that discussion aside, given that I’ve never used test scores to argue for school choice. Indeed, as I put it nearly a decade ago in “Does School Choice ‘Work’?”, “The mixed findings suggest that simply legislating ‘school choice’ programs, or enrolling a child in a charter school, will have no obvious short-term impact on achievement.” I feel like I’ve been pretty consistent, given that I observed in that essay:

For the past 20 years, the question of whether school choice ‘works’ has been understood to mean simply whether a school-choice program boosted reading and math test scores in a given year . . . Particularly problematic is how this way of thinking has caused school-choice proponents to ignore crucial questions of market design and implementation.

Third, the Louisiana program is distinctive in the substantial regulations it imposes upon participating private schools—including the requirement that they employ an open admissions policy. A crucial question is whether well-meaning regulations deter quality schools from accepting voucher students. For instance, Wolf et al. note:

To participate in the [LSP], private schools must meet certain criteria related to: enrollment, financial practices, student mobility, and the health, safety and welfare of students. Participating schools are prohibited from being selective in their enrollment of voucher students and must administer the state’s accountability tests annually to voucher students in grades 3 through 8 and grade 10.

Anyone who has spent much time in or around schools knows that even seemingly innocuous criteria relating to “student health” or “safety” can entail onerous paper chases, complicated compliance, intrusive micromanagement, and disruption. This is why well-meaning rules may deter high-quality schools from participating—especially when those schools find it easy to fill up their seats without accepting voucher students. When that happens, the participating schools will tend to be those most desperate to add enrollment. If that’s what is leading to lousy performance, one conclusion is not that the scholarships were bad for kids—but that scholarships only help when kids can use them to attend good schools.

This all brings us back to something I wrote about last week, which is why obsessing on “What works?” can do more to obscure than illuminate why and how things work. My hat is off to Wolf and his colleagues for their invaluable work, but we would do well to recognize that their findings raise at least as many questions as they answer.

Frederick Hess is director of education policy studies at AEI and an executive editor at Education Next.

This post originally appeared on Rick Hess Straight Up.

Source: EducationNext