UPDATED: A long-awaited study that CPS touted as proof of the success of turnarounds was released Wednesday–and instead showed only a small amount of progress and added fuel to calls for the district to back off.
This morning, a coalition of Local School Council members, backed by, Chicago Teachers Union announced that they plan to file a lawsuit asking for an injunction to stop the district from moving ahead with plans for 10 turnarounds, including six at schools that would be handed over to the non-profit Academy for Urban School Leadership. The crux of the lawsuit is that state law requires local school councils to be given specific action plans before being turned around. The LSC members from the targeted schools say they were never given such directives.
The CTU has long fought against school closings and pointed out they result in the firing of mostly black teachers. One finding of the study by the Consortium on Chicago School Research supports that claim: At turnaround schools, the number of black teachers plummeted, while the number of white, less experienced teachers increased.
But the study provides only a little evidence that the upheaval and financial investment in turnarounds is worth the substantial cost, estimated to be at least $20 million in the coming school year. That does not include millions more in capital spending on building renovations at these schools.
In fact, the study draws no conclusion about whether turnarounds are the best strategy to improve the lowest-achieving schools, or whether the Academy for Urban School Leadership is the best organization to carry it out.
“That is not what we were studying,” said Elaine Allensworth, a senior director and chief research officer at the Consortium.
The study found that student achievement improved at schools that underwent particular types of intervention. But the study made no distinction between three radically different strategies: schools that were turned around; schools that were closed for low achievement and re-opened as new schools, with a mostly-new group of already higher-performing students; and schools whose principals were sent to a two-year professional development program at the University of Virginia.
The only school-by-school performance data provided indicates that, compared to similar low-performing schools, AUSL’s turnaround schools moved their students ahead at a slightly faster pace. But the improvement was small, and one expert questions its significance.
Allensworth and co-author Marisa de la Torre said the growth of AUSL students is statistically significant, but refused to attach any adjective to the improvement and declined to provide an example of how much actual test scores differed or changed between the turnaround and comparison schools. (Instead of actual test scores, the study relied on standard deviation of test scores.)
“All I can say is that for some people the differences will be big, for some other people not so much,” Torres said.
CPS Chief Education Officer Noemi Donoso said the fact that the turnaround schools did better, regardless of how much better, was enough for her.
“The alternative is the status quo,” she said. “If the choice is between flat-lining and closing the gap, I would rather close the gap.”
But education experts consider the statistical figure of average yearly growth by turnarounds cited in the study as evidence of improvement to be relatively small.
“0.07 is a pretty small effect,” said Dan McCaffrey, a statistician at the Rand Corporation, a not-for-profit research and analysis organization. “It doesn’t mean that it is not meaningful, but it is small.”
McCaffrey also criticized the study’s inclusion of schools that underwent drastically different interventions. The schools that were closed and re-opened as charters and magnets got virtually all-new students, who were higher-performing when they enrolled than the students who were in the school previously, according to information in the study.
“Shifting students and changing labels is not a legitimate way to improve a school,” said McCaffrey.
A student shift, though much smaller, could be at play in some turnarounds. At Morton and Howe —the two highest-performing AUSL turnaround schools—students at the schools in the fall of the first year of the turnaround had significantly higher reading performance than students from the prior, according to the Consortium. These two schools also experienced enrollment increases, which seems to suggest that better-performing students were attracted to or recruited to attend.
Challenge from the feds
Even before the Consortium released the study, it had already been challenged. The study was funded by the Institute for Education Sciences, the federal agency that oversees education research and reports education statistics. But Rebecca Maynard, a commissioner at the agency, refused to approve the final version. (The Institute is headed by John Easton, the former director of the Consortium.)
Allensworth said Maynard wrote a short explanation that indicated she didn’t agree with the methodology.
But Maynard said she was worried that the study was drawing a conclusion without enough information. “Parts of the report were written in a way that could suggest the study was intended to answer more complex questions than was judged to be possible with the available data,” she said in an e-mail.
The Consortium analysis compared a school that underwent an intervention to a single other school with similar demographics and performance. With such a small sample–44 schools in all–other factors could have affected whether or not schools improved, such as principal turnover and extra resources.
Donoso acknowledged that schools often wind up making trade-offs because of scarce dollars, choosing between hiring extra security or paying for after-school programs. But, she said, that is the choice of the local school council and the principal.
The study also looked at the progress of two initiatives to improve high schools. One, reconstitution, was a process that mirrored turnaround in that staff were replaced. According to previous research, that strategy failed.
And the first results from turnaround high schools are not good. Because the first turnaround high school was opened in 2009, no test scores were available for inclusion in the Consortium study. However, the attendance rate increased in the first year–then declined after that.
In addition, there were only small improvements in the freshman on-track rate, which takes into account absences and grades in freshman year and is considered a strong predictor of graduation–or dropping out.