School Voucher Study Suffers from Design Flaws and Questionable Conclusions

• Duke Voucher Study critical of Opportunity Scholarship Program.
• Study acknowledges difficulties in making legitimate comparisons to students who remained in public schools; but made them anyway.
• Incorrectly assumes more accountability equates with better academic results.

Last week Duke’s Children Law Clinic released its study of North Carolina’s Opportunity Scholarship Program (OSP), School Vouchers in North Carolina: The First Three Years.  Not surprisingly, the study was highly critical of OSP for a supposed lack of accountability and poor results. While the Duke Children’s Law Clinic’s study is receiving a lot of publicity, its design and conclusions are flawed and its statements should not go unchecked. Let’s review several of the bigger problems.

In the Executive Summary, the report makes the following statements:

Based on limited and early data, more than half the students using vouchers are performing below average on nationally-standardized reading, language, and math tests. In contrast, similar public school students in NC are scoring above the national average.

 Then in making those claims the report later acknowledges in the section titled Public School Comparison that no “apples-to-apples” comparison can ever be made between voucher students and public school students.  The report continues:

Because the law allows the private schools to select their own tests, requires only a very small percentage of the test scores to be made public, and allows the public data to be reported only in aggregate form, no accurate comparisons can be made. The law requires the SEAA to report on the “learning gains or losses” of the voucher students and “compare, to the extent possible” with the learning gains or losses with similar public school students. Due to the nature of the data that will be produced by the private schools – which will never be comparable to public school data – it is unlikely that any truly valid comparison will be possible.

Yet, despite the cautions of the report’s own words, and its use of samples representing only 14 percent and 22 percent of voucher students, the comparisons were made.

I also question the use of National Assessment of Educational Progress (NAEP) data. Why not use state accountability models that tests all students?  NAEP data is from a small sample. In addition, NAEP national test averages are based on a representative sample of public, private, Bureau of Indian Affairs and Department of Defense Schools.

You’re right it’s not “apples-to-apples.”  So why try to make the comparison when you know it’s not legitimate?

A second problem concerns part of the report’s conclusion:

The research of programs from other states is nearly unanimous in showing that students in voucher programs do not have better educational outcomes than children in public schools. Strikingly, all of these studied programs have even more oversight and accountability measures built into their design than does North Carolina’s. Thus, it seems highly unlikely that the program in North Carolina will produce different and better results than the ones produced around the country.

This statement is simply not true. A casual Google search directs us to a variety of studies that highlight the benefits of vouchers.  A quick review refutes the study’s claim:

• A group of researchers from the University of Arkansas constructed an analysis of 19 research studies examining 11 school choice programs and found that there were “statistically significant achievement effects” related to “using school vouchers.”

• In a study published last year, Patrick Wolf of the University of Arkansas found that voucher recipients in Washington, DC had graduation rates of 91 percent. That’s significantly higher than the D.C. public school average (56 percent) and the graduation rate for students who applied for a D.C. voucher but didn’t win the lottery (70 percent). In addition, after three years Wolfe found there was a “statistically significant positive impact on reading test scores” but not on math test scores.

• In Win-Win: The Empirical Evidence on School Choice, Greg Forster examined 18 studies on achievement of school choice participants. Forster found 14 of the 18 programs had positive impacts. Of the 33 studies on school choice program’s effect on public school’s academic outcomes, 31 found that choice actually worked to improve the public schools.

A number of studies of voucher programs in North Carolina have also found positive impacts.

• An August 2000 review of the Charlotte Children’s Scholarship program by Jay Greene found that low-income predominantly African-American scholarship voucher recipients had combined reading and math scores six percentile points higher than the control group after only one year of schooling.

• Seven years later in a follow -up study, Joshua Cowen found similar gains for recipients of vouchers from the Children’s Scholarship Fund. More recently researchers David Deming, Justine Hastings, Thomas Kane and Douglas Staiger evaluated the relationship between school choice and college attainment and found that choice recipients from low-quality neighborhood school zones were 8.7 percentage points more likely to graduate from high school, 6.6 percentage points more likely to attend a four-year college and 5.7 percentage points more likely to earn a college degree than those not chosen by lottery to participate in the program.[i]

Moreover, consider the wholly unsubstantiated and fallacious inference in: “Strikingly, all of these studied programs have even more oversight and accountability measures built into their design than does North Carolina’s.”   That implies that the greater the oversight and accountability of the voucher program, the better the results.  It’s an assertion with no validation. It only reveals the authors’ liberal-progressive bias and the belief that the only acceptable accountability is top-down. It discounts all other forms of accountability – such as parental, institutional, collegial. For example, a parent or child’s choice to attend a school is in and of itself a form of accountability. The parents involved in selecting which school a child attends are exercising a form of accountability that doesn’t show up in the normal channels.

Lastly the conclusion to School Vouchers in North Carolina: The First Three Years states:

….. The program has no checks to protect children from the choices of their parents, which could include the choice to send a child to a fringe school that does virtually nothing to prepare a child for active participation in our democratic society after graduation, or may even undermine such participation. While surely most parents will not choose such an outcome, that such an outcome is supported by taxpayer resources is profoundly problematic.

This statement is particularly troubling. While one may hypothesize endlessly about parents making the “wrong” choice, no consideration is given to the fact children get placed or assigned to the “wrong” public school or a failing public school all the time, simply because they don’t have the resources or live in the right location to attend a school that fits their needs.

If you are worried about a “fringe school that does virtually nothing to prepare a child for active participation in our democratic society after graduation,” you’re assuming that parents or educators are unable to see the problem and take action. That is a highly unlikely scenario. If there are schools that are truly doing “virtually nothing” to prepare a child for active participation our democratic society, wouldn’t we expect that parents, educators, citizen’s, politicians and others would know about it as well and initiate action? Arguing against schools that do “virtually nothing” to prepare their students is a straw man argument. Who would be against such a proposal? But also, there isn’t any evidence that such schools even exist.

School Vouchers in North Carolina: The First Three Years is a report highly critical of the OSP program. A closer look reveals the report and conclusions to be flawed with a poor design and muddled conclusions. It’s a document that propels advocacy at the expense of research and should be treated as such.

[i] For an extended discussion on the research and benefits of school choice, see: First in Freedom: Transforming Ideas into Consequences for North Carolina, John Locke Foundation, See: Chapter 3, Parental School Choice, Appendix A: Research on the Benefits of School Choice, (Author: Terry Stoops, Ph.D.).