I wanted to make one additional point about the CREDO study I mentioned in the previous post, but I’m putting it in this separate post because it is a more boring econometric question that I don’t want to bog down the other more obvious issues with.

There is an important assumption the study makes and people should ask themselves how believable it is. The assumption is this:

Take a group of, say, six students from the public school system who have the same grade level, gender, race, income, special ed and english language learner status, and previous achievement test scores. One of these students decides to go to a charter school, and the other five do not. It is assumed that the one who chose to leave would have performed the same as the other five students had he stayed in the public school.

The presumption here is that there is no systematic difference between these students that is not captured by these variables. This begs the question, if these students are the exact same, then why did one decide to go to a charter school and the other five didn’t? Obviously there are some unobserved (to this study) variables that explain the decision to attend the charter school. Is it believable that these unobserved variables are uncorrelated with the student’s performance in school?

To illustrate the assumption even more starkly, imagine that the child goes to a virtual charter school. Is it believable that the kid whose parents pull him out of school to have him go to school on the internet is not different than the comparison kids? The importance of unobserved variables here suggests to me that these results will be biased against charter schools.

I’m not saying the assumption the authors have made is necessarily false, but simply that people who tout this study should understand that this assumption underlies the results, and they should ask themselves whether in another context they would consider it believable. I would argue that had the results of the study shown that on average charter schools outperformed public schools, some charter school critics would be dismissing the study on the basis that charters are cream skimming on unobservable variables, e.g. the charters are accepting systematically smarter, better performing students. Then again, perhaps I am biased and would be less skeptical of the assumption in this case.

My best guess is that whether or not this assumption holds depends, and that in some schools or states it does and in others it doesn’t. In Ohio, where there are a large number of virtual schools, I’m guessing it doesn’t. In New York, where charters are generally high quality and whether or not a student gets into onel contains a large randomization component, the assumption probably does hold. In fact, CREDO was able to replicate the results of a study based on randomization using their matching technique, which suggests the assumption does hold in New York.

So take it as you will, but understand that this assumption is there, and it’s important.