I’ve defended KIPP before from accusations that their performance is due to selection bias and cream skimming. A new paper from Josh Angrist and a bunch of other co-authors provides more evidence in KIPP’s favor. They found that KIPP benefited the weakest students and those with special needs most:
Our results show average reading score gains of about 0.12 standard deviations (hereafter, σ) for each year a student spends at KIPP, with significantly larger gains for special education and LEP students of about 0.3-0.4σ. Students attending KIPP gain an average of 0.35σ per year in math; these effects are slightly larger for LEP and special education students. We also produce separate estimates for students with different levels of baseline (4th grade) scores. The result suggests that effects are largest for those who start out behind their peers. Male and female students gain about equally in math, while boys benefit more than girls in reading. Finally, an examination of Massachusetts Comprehensive Assessment System (MCAS) performance categories (similar to quartiles) shows that KIPP Lynn boosts achievement primarily by moving students up from the lowest group. Together, therefore, the findings reported here suggest that KIPP Lynn benefits weak students the most.
The study also provides further evidence against the claim that KIPP’s model is based on kicking out the trouble students. The authors found that students who won the admissions lottery were no more likely to switch schools than those who lost, meaning that controlling for selection bias, KIPP students do not have a higher attrition rate.
Initial skepticism of KIPP’s high achievement results was understandable, and questions about cream skimming, kicking out bad students, and whether the weakest students were benefitting were important to ask. But the evidence continues to come in that KIPP is doing something right, and that skeptics were wrong about why that was. It’s time for skeptics to re-evaluate their positions. I’m hoping that when Diane Ravitch releases the “The Death and Life of the Great American School System” in paperback the section that is skeptical of KIPP is updated to reflect this new evidence.
30 comments
Comments feed for this article
Thursday ~ June 2nd, 2011 at 9:05 am
Th
“The authors found that students who won the admissions lottery were no more likely to switch schools than those who lost, meaning that controlling for selection bias, KIPP students do not have a higher attrition rate.”
Obviously the people who enter the lottery are dissatisfied with their current school and are looking for something better. The ones who lose the lottery are still in this situation and would continue to look at options. Attrition rate at the KIPP school should be significantly less than lottery losers unless the critics are correct that KIPP kicks out their problems or that parents quickly become disillusioned with KIPP.
Thursday ~ June 2nd, 2011 at 9:21 am
cdb
It’s so frustrating to have so much energy spent debating/defending GOOD schools instead of figuring out how to fix BAD ones.
Thursday ~ June 2nd, 2011 at 9:27 am
Gepap
To add to what Th said, signing up for a lottery in an of itself is a selection mechanism – it filters out those parents not involved enough to even sign up unless the lottery is universal (ie. it covers all children within some catchment area).
I have no doubt that KIPP’s formula works by itself Its pretty common sensical. The problem is upscaling the KIPP model universally, to include children whose parents could not be bothered, for example, to sign up for the KIPP pledge.
Thursday ~ June 2nd, 2011 at 9:35 am
Adam Ozimek
The lottery doesn’t just filter out parents not involved enough to even sign up, but it filters out parents who are satisfied with their existing public schools, whose children on average have higher grades than KIPP applicants. The KIPP model doesn’t need to be universal in order to be extremely useful. It can on the one hand improve educational outcomes for some students, and on the other hand put pressure on public schools to improve. Other successful charter school models can do the same, not just the KIPP model. And I’m not so much of a public school pessimist as you to think that the only possible result of more competition is that they fail completely and go extinct.
Thursday ~ June 2nd, 2011 at 8:49 pm
Eric Morey
Adam,
My concern is that as the case for the value of KIPP strengthens, there will be more and more pressure to roll it out inappropriately in a way that doesn’t scale to school systems that won’t benefit or will be worse off.
Friday ~ June 3rd, 2011 at 10:01 am
Gepap
The point of public education is to make an education available to ALL – universality is the key. If the KIPP model can’t be replicated succesfully with a population with greater outside problems than those of KIPPs students, then how can the “pressure” it supposedly places on those schools that have to deal with this population be constructive? Because the point supposedly is to provide a better education to all kids, to give them the ability to rise above circumstances not of their choosing (like their family).
The point of charter schools was pedagogic experimentation, which is a good thing, and good for KIPP for showing one successful approach. The problem is not KIPP, but instead this notion of “school reformers” that somehow teachers can solve the failures of our society to tackle poverty and joblessness. A good teacher is important, but they are not a panacea for our broken down system. Greater early childhood education and more social transfer spending would likely raise educational outcomes more than if every school was a KIPP school.
Friday ~ June 3rd, 2011 at 5:46 pm
Adam Ozimek
“if the KIPP model can’t be replicated successfully with a popularion with greater outside problems than those of KIPP’s students…”
If seen no evidence whatsoever that inability to deal with students with “greater outside problems” is what would prevent KIPP from being replicated. The point of this study is that KIPP helps the worst students most, which is more the opposite of what your saying than anything else.
Since KIPP doesn’t crowd out early childhood education or more social transfer spending, I’m not sure how your last argument cuts against it. Also, more social transfer spending is not a panacea either, and it can’t solve the failures of our society or tackle poverty or joblessness. There, now we’ve got two straw men out of the way….
Friday ~ June 3rd, 2011 at 8:55 am
Winners and Losers in the Charter School Lottery | SA Griffin Blog 3
[...] Adam Ozimek points today to a study of KIPP charter schools that finds good outcomes for KIPP students and concludes that none of it is attributable to “skimming.” That is, it’s not the case that KIPP schools are getting good scores because poor students are prodded to leave at higher rates than good students. [...]
Friday ~ June 3rd, 2011 at 6:18 pm
Th
“Since KIPP doesn’t crowd out early childhood education or more social transfer spending”
You’ll be eating those words within 5 years. We were told the same about Medicare Advantage.
Sunday ~ June 5th, 2011 at 10:07 pm
govt_mule
Adam – in your 3/18/10 and 4/12/10 posts you examined studies that were not favorable to KIPP closely. You point out that study quality matters, and that one shouldn’t blindly cite results without examining the quality and rigor of the data and methodology. Among other (appropriate) criticisms, you mentioned “negative results … based on a pooled national average, which I’m not sure is all that informative”.
Please apply the same careful review to the pro-KIPP studies you cite.
You’ll find numerous flaws in the Wolfram study you cited 4/19/11, including the use of pooled results that conceal much of the information that could be extracted from the data. Did the schools with the highest performance also have the highest attrition rate, highest 4th grade scores, or number of students repeating grades? You’ll never know. You’ll find that the claims about attrition being the same at KIPP and public schools to be complete baloney in any meaningful sense, though you’ll have to extract the actual attendance figures from the data yourself to discover this. Also note that this report was paid for by KIPP and was not peer reviewed.
The report by Angrist et al. cited 6/2/11 is also weak. “Since we have the most outcome data for the earliest admitted cohorts, the results in the paper should be interpreted as the eff ect of a relatively new school on its fi rst cohorts of students.” Only 138 kids were in the first cohort and they had only attended KIPP for an average of 2.6 years. I don’t think you can draw any meaningful conclusions about KIPP from a study of a few hundred kids at a single school. Do you?
There have been several reports showing that weaker students improve at KIPP schools, while stronger students actually do worse at KIPP than in public schools. Do you have any hypothesis to explain this puzzling observation?
Monday ~ June 6th, 2011 at 7:22 am
Adam Ozimek
KIPP schools are both a coherent group and are run in a standardized fashion, so I don’t suspect heterogeneity would be a problem. Furthermore I don’t see any competing sub-total pooling that could be used, and nothing you propose is an alternative. So I don’t see any reason to question the pooling result.
You may want to see further results, but that’s not really a criticism of the results they presented. The question here is KIPP performance relative to the comparable local schools, and everything your posing looks like explaining between KIPP school performance, so I honestly don’t see the relevance of the questions your posing for the question at hand. These aren’t “hidden” in the pooled analysis, but rather would require entirely different analysis.
Please explain how the results are baloney if you extract actual attendance figures from the data.
The results in Angrist are from a small sample size, but it is a rigorous methodology and the results are informative and meaningful. I wouldn’t hang my hat on these results alone as demonstrating attrition is not a problem among all KIPP schools, but the combination of this and the previous evidence together does constitute strong evidence.
Tuesday ~ June 7th, 2011 at 10:57 pm
govt_mule
” I don’t see any reason to question the pooling result.”
Again, you complained in your 3/18/10 post about the CREDO study that use of pooled national was inappropriate for comparing public and charter schools. Are you now saying that pooled national data is ok?
Wednesday ~ June 8th, 2011 at 7:17 am
Adam Ozimek
I”m puzzled by your complaint here. Yes, for all of the reasons I explained it makes sense in this context. Did you take my point to be that you should never use pooled national data?
Furthermore, I didn’t say you shouldn’t pool the results in the previous post, I just said that it wasn’t all that informative because state level results were much more informative. I make a specific argument for a better grouping of the data, which was done, and I’m saying is a more informative measure of charter performance. That has no relevance for this example.
Monday ~ June 6th, 2011 at 5:44 pm
govt_mule
“KIPP schools are .. run in a standardized fashion..I don’t see any reason to question the pooling result.”
It don’t see how you can support the contention that KIPP schools are homogeneous. Figure ES.5 (Tuttle et al. 2010 ) shows that test score “effects” (KIPP vs Public) range from -0.03 to 0.81 in math and 0.11 to 0.91 in reading for the 22 schools studied, with several schools having about double the mean math effect and one school having double the mean effect in reading. If that happens to be the Lynn school studied by Angrist, then their results are not at all representative of the “standard” KIPP facility.
Information was not hidden by pooling, but by ordering each data set differently so that correlations between factors and outcomes could not be determined. Had the authors simply plotted each data set in the same school order, the reader could readily see how baseline test scores, special ed and language classification, attrition rates, repeated grades, etc. correlated with outcomes. There’s a wealth of hidden information about what factors might account for the very wide distribution in performance of KIPP schools which requires no additional analysis to reveal, just a small change in the way the data are plotted.
The Mathematica paper devotes a lot of attention to attrition, claiming that rates are similar in KIPP and public schools. They use a very narrow and misleading definition of attrition, however, treating transfers between (equivalent) public schools the same as transfers from KIPP to public schools. Meaningful attrition levels can be determined from enrollment figures taken from Table II.2. The average fractional enrollment for grades 5, 6, 7, and 8 is 1.00, 0.82, 0.53, 0.36 for KIPP and 1.00, 0.95, 0.91 and 0.89 for the public schools. KIPP schools lose over 60% of their students by the end of grade 8, public schools lose only 11%.
Tuesday ~ June 7th, 2011 at 7:28 am
Adam Ozimek
They are clearly homogeneous enough to be considered a coherent group about which one could take a mean. What is the sub-group your proposing that makes more sense? Or are means just invalid in this context? If you can’t consider these a coherent and useful group then you should be prepared to reject all means and group comparisons in education research.
And again, you may be concerned about the what explains the performance of KIPP schools relative to each other, but again the question is whether attrition is higher for KIPP than at comparable schools.
I don’t see the table II.2 you’re referring to.
Tuesday ~ June 7th, 2011 at 9:20 am
govt_mule
page 18. “Table II.2. Grade Repetition Rates by KIPP School and Grade Level”. Or use any of the other tables that give “N = XXX” values to determine actual enrollment.
Monday ~ June 6th, 2011 at 10:21 pm
Stuart Buck
“KIPP schools lose over 60% of their students by the end of grade 8, public schools lose only 11%.”
I don’t think that can possibly be true. In the Mathematica attrition study (http://www.mathematica-mpr.com/publications/pdfs/education/KIPP_middle_schools_wp.pdf), go to Table A.5. The last column shows the “the cumulative probability that a student in fifth grade will leave that school before completing eighth grade.”
In NO KIPP school was the attrition rate as high as the cumulative 60% that you claim as the average. The cumulative attrition rates ranged from 11% to 50%.
Tuesday ~ June 7th, 2011 at 10:04 pm
govt_mule
“In NO KIPP school was the attrition rate as high as the cumulative 60% that you claim as the average”
True. I failed to weight the data I used by school enrollment before averaging. Doing so gives cumulative average fractional enrollment of 1, 0.86, 0.62, 0.49 for KIPP schools and 1, 0.98, 0.99, 0.99 for public schools.
My numbers are also different because I’m using different methodology and a different set of data.
But whether average KIPP attrition is 51% or 34%, public school attrition is in no way comparable. Claims that 35% of urban public school 8th graders (13-14 year olds) have left the public school system are baloney.
Wednesday ~ June 8th, 2011 at 7:45 am
Adam Ozimek
The relevant comparison is school-to-school, not school-to-district, especially given that KIPP students are a higher percentage minority and lower achievement than the districts overall, but also for all of the very obvious reasons why you should compare schools to schools instead of schools to districts. There is no way you would argue for the comparison you’re demanding except that you’re grasping for anything to try and reinforce your preconceived biases. I appreciate that you are digging in to these studies and taking an interest in the issue, but I’m not going to keep debating this with you, because you’re not able to see what is obvious here, so there’s no point to it. I’m not going to debate with you the molecular composition of air if you can’t see that the sky is blue.
Wednesday ~ June 8th, 2011 at 11:47 am
govt_mule
“The relevant comparison is school-to-school”
You’re certainly entitled to your opinion that a charter school that is open to any student in an large urban school district should be compared to a single school in that district. However, the Mathematica study compared performance of the KIPP school with the district as a whole – either using all district students for the Districtwide OLS approach or a districtwide matched comparison group : “The comparison group is carefully selected by considering all students across the district” (Tuttle et al., 2010, p20). Only in the attrition aspect of the study was the comparison unit changed from district to individual school. This seems to me an artificial distinction intended to generate misleading data regarding the extent of attrition in the public school system.
“There is no way you would argue for the comparison you’re demanding except that you’re grasping for anything to try and reinforce your preconceived biases”
??
Wednesday ~ June 8th, 2011 at 2:24 pm
Adam Ozimek
Ok, I know I said I was done debating this, but one more comment: If you’re going to talk about attrition out of something, then you need to look at it at the school level. Comparing out-of-district migration to out-of-KIPP migration is silly, which is what you’re doing by looking at district-wide sample size. In the attrition section of the first paper they in fact do it all three ways: districtwide, matched, and feeder school. Look at figure I and read this “In Figure 1 we present the full attrition results, averaged across sites, for KIPP students, the district comparison group, and the full district sample”….. ANd “Over the entire course of middle school, cumulative attrition rates at KIPP schools in our sample are similar to those of schools in their surrounding district, on average. In the average site, the attrition rate at KIPP is 34 percent, compared with 33 percent in the district comparison group, and 35 percent in the district as a whole.”
Wednesday ~ June 8th, 2011 at 5:32 pm
govt_mule
“If you’re going to talk about attrition out of something, then you need to look at it at the school level”
Let’s use a concrete example to see if I can understand your viewpoint. Gotham has 26 schools: 23 public schools A – W, two KIPP schools X and Y, and a parochial school Z. The KIPP schools, like the public schools, are managed similarly, but each school has some unique traits – some offer track, some offer basketball, some offer Spanish, some offer more support for special ed, etc. Kids transfer from school to school based on these traits, as well as for other reasons (e.g. to avoid bullying, to shorten the bus ride after they move).
I think we both agree that a student who transfers from KIPP to a public or parochial school should definitely be counted as attritted, as should a student moving from a public to a private or charter school.
The question is whether transferring a student from KIPP school X to KIPP school Y is considered attrition from the KIPP program. I think it’s silly and misleading to treat a transfer within the same education system as attrition when we are comparing KIPP schools to public schools.
The same is true of public school students. Simply transferring from building C to building N in the same public school system is not attrition in any meaningful sense and should not be counted as such.
This doesn’t change if there is only one KIPP school per district. Transfers between educational systems should count as attrition, transfers within a single system should not.
Questions:
Should transfers between KIPP schools in the same district be considered attrition?
Should transfers between public schools in the same district be considered attrition?
Wednesday ~ June 8th, 2011 at 9:40 pm
Adam Ozimek
Yes and yes. Let me provide an example that should make it even clearer why. Say there are 10 public schools in a district, all with the same performance, and same number of students leaving each year to go to another school within the district, and the same number leaving to leave the district by moving far away. Now say one of them is converted to a charter school, but the behavior of students is identical. Using your measure, attrition at the public school converted to a charter skyrockets and will be well above the rest of the district. I think it should be pretty clear if one wants to understand performance of the charter it makes no sense to use the measure you’re suggesting.
Wednesday ~ June 8th, 2011 at 11:20 pm
govt_mule
” Using your measure, attrition at the public school converted to a charter skyrockets”
No, I don’t believe it would.
Say each of the ten schools is at steady state with 200 kids per grade per school. Each year 5% of the kids in grades 5 – 8 transfer to another district public school, and are replaced by an equal number of kids transferring from the other 9 schools. 5% move out of district and are replaced by an equal number of kids moving in from other districts. According to Tuttle et al. there is an attrition rate of 10% per year or 40% cumulative. According to me there is an attrition rate of 0, because I don’t care what individual students do, I care about total enrollment changes from grade to grade. Individual students come and go for many reasons, but even a median quality school will be able to replace kids who leave with new students.
Say school 5 gets a new principal who’s a real jerk and alienates students and parents. 20 kids a year continue to transfer out, but no one transfers in any more, and enrollment for grades 5 – 8 goes to 200, 180, 160, 140. From the Tuttle perspective the attrition rate is still 10% a year. From my perspective this school’s attrition rate has changed from 0 to10% a year, while the other district schools would have a slight negative attrition rate from picking up the extra pupils.
If one school becomes a Charter but keeps its enrollment at 200 per grade (i.e. student behavior is unchanged), then from my perspective attrition is still 0 (and 10%/year by Tuttle). If it ends up with the enrollment pattern like s chool 5 above, then it would have an attrition rate of 10% a year by both my and Tuttle’s criteria.
Thursday ~ June 9th, 2011 at 7:15 am
Adam Ozimek
This is not attrition you are looking at, and is not what people are complaining about with KIPP, and it is an entirely separate issue. You’re looking at enrollment, nothing else. You may think that enrollment is important, but that’s not the issue at hand. If Mathematica had addressed enrollment instead of attrition everyone but you would be asking “why are they looking at this issue instead of attrition?” and they would have been rightly criticized.
If you are concerned about enrollment by looking at the sample sizes, which it seems you are, keep in mind that the many of the KIPP schools are young, and are in stages of increasing enrollment so later cohorts could be smaller than younger ones over time for that reason as well.
Thursday ~ June 9th, 2011 at 1:02 pm
govt_mule
“This is not attrition you are looking at, and is not what people are complaining about with KIPP”
It is attrition and it is exactly what people are complaining about.
The complaint is that KIPP schools take in X number of kids in 5th grade and then weed out the ones who don’t test well, while the public schools have to keep all the poor performers. This deliberate dumping of students from the KIPP system into the public system is what we want to detect (if it exists), not normal fluxes of students between equivalent schools.
One detection approach (Mathematica) is to measure the flux of students out of each grade and school and look for anomalies. But in large urban school districts with a fairly high “background” flux, dumping is hard to detect. As shown in the Gotham hypothetical, both the “good” schools which maintain stable enrollments and the school with the crappy principal that loses 10% of entering students each year, have the same attrition rate according to the Mathematica approach. This approach is not able to detect dumping unless it produces fluxes much higher than the background.
A second detection approach looks at steady-state input and output flux, or equivalently, the change in enrollment from grade to grade within a school relative to the grade to grade enrollment in the general student population. This enrollment approach readily detects the loss of students at the hypothetical crappy school, although it could be confounded if the crappy school was able to recruit enough new students to replace the dumped ones. It could also give inaccurate results if the school is not at steady-state, e.g. is expanding lower grade enrollment before upper grade. But that effect can easily be adjusted for by including such readily-available information in the calculations
w
Friday ~ June 10th, 2011 at 7:49 am
Adam Ozimek
Read Ravitch, for example. Her complaint is about the exit rate, not the net exit rate. Also, KIPP schools are nowhere near the steady-state, nor do you have any information to conclude that the districts are at a steady state rather than having a steadily increasing population.
Friday ~ June 10th, 2011 at 3:31 pm
govt_mule
The fundamental disagreement seems to be over the meaning of “exit”/”attrition” in a study of two treatments. Say we are comparing methadone treatment vs psychotherapy for heroin addiction and want to account for variation in treatment exit rates as a potential bias in effectiveness. In this context, “exit”/”attrition” should be defined as either switching to the other treatment program or stopping treatment altogether, not transferring internally from one methadone clinic to another, or one psychiatrist to another.
I agree KIPP enrollment may not be at steady state but there is sufficient data to account for that. Public school enrollment seems pretty steady. I’ve put the the extracted data here:
https://spreadsheets.google.com/spreadsheet/ccc?key=0AmX3_b8dC–cdGIySXhrOE5FM0VybDdwamxob1d0R3c&hl=en_US&authkey=CLua29oK
Sunday ~ April 21st, 2013 at 9:13 am
Big Time Self-Plagiarism . . . Part 2 – @ THE CHALK FACE knows SCHOOLS MATTER
[...] successful charter networks is the Knowledge Is Power Program and the latest research (PDF, via Adam Ozimek) once again shows substantial KIPP-linked gains for poor kids, especially the weakest students and [...]
Sunday ~ April 21st, 2013 at 9:14 am
Big Time Self-Plagiarism, Punked Web Lemmings, and Chain Gangs for All? Part 1 – @ THE CHALK FACE knows SCHOOLS MATTER
[...] report. Drum, as well as Iglesias and Kain, reference economics consultant and KIPP enthusiast, Adam Ozimek, whose enthusiasm to have Diane Ravitch get on the KIPP bandwagon will apparently go to any [...]