Chicago Public School officials and Mayor Richard Daley incited a barrage of media ridicule when they touted the district’s dramatic state test score gains as “historic” without acknowledging that revisions in the test might have contributed to the increase.
Publicly, CPS is defending the validity of its gains, which are in the double-digits in almost every grade and subject. In mid-July, an indignant Arne Duncan appeared on WTWW-Channel 11’s “Chicago Tonight” public affairs show after a newspaper editorial mocked the district’s “fuzzy math.” Duncan insisted that statewide results, which will likely be released later this summer, would show Chicago’s gains dramatically outpacing the state. “It’s as if people don’t want us to succeed,” he complained.
But privately, the district is working with the Illinois State Board of Education to get to the root of Chicago’s surprising achievement.
“People have so many hypotheses about why this came about, chasing them down is a full time job,” says Rense Lange, a testing expert who works in the state board’s assessment division. “Chicago likes to go up but they don’t want to go up for the wrong reasons.”
State board officials still defend their own assessments as reliable and Chicago’s gains as valid. New classroom assessments tied to state standards likely sharpened teachers focus and helped boost achievement, says Daniel Bugler, who oversees testing and accountability for CPS. By early August, the state’s analysis of Chicago’s scores had not revealed any patterns that would make the gains suspect, he adds.
Outside testing experts remain skeptical about the gains. “It could be that these kids are taller, it could be the ruler is different,” quips Timothy Shanahan, director of the Center for Literacy at the University of Illinois at Chicago.
Some other large or low-income districts are also finding sizable gains in their preliminary data. Peoria reports increases between 2 and 20 percentage points in every grade and subject, somewhat lower than Chicago’s 9 to 33 percentage point increases. Posen-Robbins School District in south suburban Chicago also noted increases, although they are still calculating their gains.
Exactly what is behind the dramatic test score gains in Chicago and possibly those in other school districts across the state? A close examination reveals that extensive tinkering by state education officials with test pass rates, format and policy likely boosted scores without raising student achievement. Catalyst Chicago identifies four of them:
1. Lower the standard. For years, educators around the state complained that the bar for passing the 8th grade math test was set too high, according to state education officials. So this year, the state agreed to lower the cut score from the 67th percentile to the 38th percentile, putting the passing score on par with those set for 3rd and 5th grades.
“This isn’t a slight adjustment, this is a big shift,” Shanahan notes. “Every district in the state will be affected”
Not surprisingly, pass rates on the 8th grade math test skyrocketed in Chicago with 66 percent meeting standards this year compared to only 33 percent last year.
2. Give extra time. For the first time this year, teachers were given the option to extend testing time by up to 10 minutes per 45 minute testing session. Educators had complained that the time pressure made some students anxious, explains Rebecca McCabe, who oversees student assessment for the state board. Research by ISAT’s new publisher, Harcourt Assessment, indicated that extra time mainly helps kids at the bottom end of the scale and doesn’t impact the percent meeting standards, she says.
But one national testing expert says overall, research is actually mixed on whether extra time boosts standardized tests scores. Jamal Abedi , a psychometrician at the University of California at Davis, says that his own peer-reviewed research for the National Center for Research on Evaluation, Standards, and Student Testing finds that extra time means more right answers, and not just for the lowest performing students.
3. Change the test format. This year’s test featured color charts and graphs and an easier-to-read answer sheet. Educators from around the state are reporting that kids responded well to the color format, says Lange of the state board’s assessment division. “They like it better and they tried harder. Having the test in color is a motivating thing.”
And the new answer sheets probably didn’t hurt either, says Shanahan “If the answer forms are clearer, you’ve got to believe that there would be fewer mistakes.”
4. Change the types of questions. This year ISBE replaced some questions that required a long written response with questions that required only a brief written response. An analysis of a sample of tests indicates that Chicago students were more likely to attempt the short answer questions on this year’s test than the so-called “extended response” questions on last year’s test, Lange reports. “In the past, you had a lot of Chicago kids who didn’t get around to answering the longer questions.”