Diane Ravitch has become the education reform movement’s Benedict Arnold. After turning against the accountability and school choice reform movement she once staunchly promoted,  she has been hitting the op-ed pages and is now taking part in a debate at TNR. Unlike some critics, both reform and anti-reform, I won’t begrudge her changing her mind. I think it took a lot of guts to do a public about-face as she has, and I think her willingness to do so lends to her credibility rather than detracting from it.

That said, I want to draw attention to one piece of evidence she that she keeps bringing up, which is that test scores are lower at charter schools on a national level:

The only major national evaluation of charter schools was carried out by Stanford economist Margaret Raymond… Her group found that compared to regular public schools, 17% of charters got higher test scores, 46% had gains that were no different than their public counterparts, and 37% were significantly worse.

I’ve said before that this surprises me, and is in contrast with much of the evidence I’ve seen, so I decided to dig up the study that she refers to and take a closer look.

The first thing to note is that the negative results are a based on a pooled national average, which I’m not sure is all that informative. As the authors and Ravitch both say, the performance of charters relative to traditional public schools varies widely in quality by state:

In five states ‐ Arkansas, Colorado (Denver), Illinois (Chicago), Louisiana, and Missouri ‐ charter school students experienced significantly larger growth… In six states — Arizona, Florida, Minnesota, New Mexico, Ohio, and Texas — charter school students experienced lower learning gains… In four states — California, District of Columbia, Georgia, and North Carolina — the results were mixed or no different…

Some of these low performing states probably have good policy-based reasons that they are low performing. In fact, the study found that “results vary strongly by state and are shown to be influenced in significant ways by several characteristics of state charter school policies”. For example, they found that a cap on the number of charter schools has a negative impact on student performance.  The lessons I would take from this are that policies matter for charter quality, the poor performing states should learn from the good, and this may mean removing the cap on the number of charter schools. The only thing this evidence cuts against is Ravitch’s straw-man argument that charters are a magic bullet that will fix everything no matter what.

Given the high variability in state performance, it’s important to take a more detailed look at state policies. Here are how Ohio and Texas did in the Stanford study:

Pretty terrible, and among the worst in the study. But a recent RAND study had this to say about Ohio’s negative charter school performance:

The dramatically lower estimated performance of Ohio’s K-entry charter schools appears to be attributable not to grade level per se but to virtual charter schools that use technology to deliver education to students in their homes. Virtual schools constitute a large part of the enrollment of K-entry charter schools in Ohio, and students have significantly and substantially lower achievement gains while attending virtual charter schools than they experience in TPSs.

Allowing kids to be home-schooled at “virtual schools” on the internet decreased their academic performance? Gee, that’s a big surprise. I don’t know if schools like this were included in the study Ravitch cites, but this should be a firm reminder of the problem of looking at charter schools as a class without controlling for schools like this.

Another problem state is Texas, about which the RAND study had to this say:

Relative to local averages, prior achievement levels of charter entrants were particularly low in Texas, which could be attributable (at least in part) to the success of the provision in the state’s original charter law encouraging the establishment of charter schools for disadvantaged students.

These pre-existing differences in students again should caution against high level studies, and studies that do not carefully control for selection bias.

I’d be willing to guess that if this national study had shown that charter schools outperformed public schools that reform critics would be protesting that the charters are probably just cream skimming, and the results can’t be trusted. And they’d have a point. Controlling for unobserved differences is really tricky. I haven’t looked closely at how this study does it, but no matter what you do you’re going to have a concern about this problem. This exactly why randomized studies are so important, and why we can have much higher confidence in their results.

One very good recent example is a study based on randomized school assignment in New York City that found drastically positive results for charter students:

On average, a student who attended a charter school for all of grades kindergarten through eight would close about 86 percent of the “Scarsdale-Harlem achievement gap” in math and 66 percent of the achievement gap in English… a lotteried-out student who stayed in the traditional public schools for all of grades kindergarten through eight would stay on grade level but would not close the “Scarsdale-Harlem achievement gap” by much.

Replicating the results of NYC and preventing outcomes like Ohio should be what people who care about students should be talking about and focusing on, not talking points about national averages that don’t address most of the important questions.

I’m not saying the study Ravitch cites tells us nothing. I just think if this the compelling study she’s going to use as a talking point and continue to cite to as an important result then I’m not optimistic about her ability to draw relevent conclusions from empirical evidence.