Teachers who score the lowest under the district’s relatively new evaluation system are overrepresented in schools with the highest concentration of poor students, according to a new report that looks for the first time at how teachers’ scores correlate with characteristics such as student poverty, teacher race and school climate.
Issued Tuesday by the University of Chicago Consortium on School Research, the report shows that 30 percent of teachers with the lowest observation scores teach in the district’s highest-poverty schools, while just 9 percent teach in lower-poverty schools. Conversely, more than a third of the best-scoring teachers work in lower-poverty schools, and just 6 percent are in the district’s poorest schools.
That means students who attend the poorest schools are not only “more likely to be taught by teachers with the bottom scores on observations, they are also the least likely to be taught by teachers who have the top scores.”
Teachers at higher-poverty schools who scored in the top 20 percent on their observations also scored “substantially less” than teachers in the top 20 percent at low-poverty schools. And differences in evaluation scores persisted even after controlling for teacher experience levels and credentials.
Researchers say, however, that they don’t know whether the discrepancies reflect true differences in the quality of teaching or that it’s just harder to be effective or get high marks in higher-poverty schools.
“They may reflect that it is harder to get high scores in unorganized, chaotic schools or in schools with few resources or instructional supports for teachers, as other studies have shown,” the report notes.
Past research has shown students in higher-poverty schools often receive more scripted instruction and test prep — instead of the kind of deeper learning that earns higher evaluation marks — and that teachers prefer to work in schools where they’re more likely to be effective.
The researchers expressed concern that the rating system could create disincentives for working with disadvantaged students and have ramifications for teachers of color.
“If minority teachers are more likely to work in contexts where it is difficult to get high ratings, the composition of the workforce itself could be affected by the personnel decisions based on evaluation scores,” the report states. “This is especially true for African American teachers, who disproportionately teach in higher-poverty schools.”
Similar findings elsewhere
Chicago’s teacher evaluation system, called REACH, incorporates multiple teacher observations and a so called “value-added” component that measures student growth on math and reading tests. In an attempt to keep a level playing field, the growth ratings are adjusted for student characteristics and prior test scores.
Chicago developed REACH to comply with a 2010 state law. Previously, the district used a 1970s-era checklist.
While the report identifies trends, it says more research would be needed to explain the causes. But co-authors Jennie Jiang and Susan Sporte say the strong correlation between poverty and teacher evaluation scores is consistent with recent findings elsewhere.
“Chicago is not the only one,” Jiang says. “You see this in many, many states and districts across the nation.”
In a written statement, CPS Chief Education Officer Janice Jackson said while REACH represents “a dramatic improvement over past evaluation systems,” the district is working to improve it.
CPS and the Chicago Teachers Union have discussed taking into consideration school context — such as poverty levels, special education needs and administrative changes — on teacher evaluations and Jackson wants to make that a higher priority, according to a recent letter she sent to the joint committee reviewing REACH.
“The challenges highlighted in this report are complex and are not faced by CPS alone, but we are working with the CTU, researchers and technical experts to better understand their root causes and address any improvements we can make to better support our teachers and students,” Jackson said in a statement.
The report analyzes teacher evaluation data from 526 Chicago schools — excluding those that are privately run — during the 2013-14 school year, the second year the district used REACH.
Researchers looked at observation data from some 18,000 teachers — both tenured and non-tenured — as well as value-added data from 4,900 teachers who work with 3rd- to 8th-graders in math and reading. (They excluded high school teachers from the value-added analysis because that part of the evaluation is still in what they considered to be a pilot phase.)
Experience, race, gender
Among the report’s other findings were:
- Teachers with more years of experience scored better on their observations and in student growth than newer ones, and teachers with credentials like National Board certification and advanced degrees did better on their observations. First-year teachers scored “significantly” lower on their observations and in student growth, and scores dipped slightly for teachers who had more than 15 years of experience.
- On average, white teachers scored better than non-white teachers on their observations. But much of that difference was due to the “substantial relationship” between observation scores and school characteristics. Teachers of color are overrepresented at the highest-poverty schools, where teachers were more likely to get lower observation scores.
- There were no significant differences by teacher race or ethnicity on the student growth components of the evaluation.
- Male teachers had lower observation and student-growth scores than female teachers with the same level of experience and credentials — and the results changed little when adjusted for the school they taught in. This finding surprised researchers at both the Consortium and the CTU.
- Teachers in schools with better organization and learning environments, as measured by the Consortium’s 5Essentials and supplemental surveys, tend to score better on observations and student growth.
When the teacher evaluation system was put in place, Sporte says, many believed teachers could be ranked without regard for where they taught.
“This work would seem to indicate that’s just not the case, that… you can’t just pick that teaching part out from that interactivity that happens between teachers and students in the class,” she says.
Carol Caref, a CTU researcher, says the report substantiates much of the anecdotal evidence the union has collected on REACH.
She says it’s not surprising that teachers at the highest-poverty schools have low ratings. In challenging schools with poor leadership, it’s harder to develop practices consistent with high ratings, she notes, such as helping students take responsibility for their own education.
“When the school is kind of portrayed as a more struggling school, that interferes with the culture and climate of the school,” she says. “You’re not teaching in a vacuum.”
While teachers and principals play a role, she says, the highest-poverty schools would benefit from accommodations like more social workers and smaller class sizes.
Findings can point toward improvements
The study didn’t look at how a teacher performed in front of different students — say in honors geometry versus a general education math class — to see if students affected the teacher’s score. It also doesn’t address possibly systemic differences in how students are assigned to teachers.
“For example, minority or male teachers may be assigned more students with discipline issues, or with previous low achievement or special education status,” the report notes. “If so, it is possible this might negatively impact what an observer sees in their classroom and therefore what score they are assigned.”
The report also doesn’t address whether the evaluator’s characteristics, for example race and gender, had an impact on the teacher’s observation. (REACH observations are conducted by trained principals and assistant principals who must pass a state certification test.)
And while the report continues to raise questions about whether observations and value-added measurements “accurately capture teacher quality,” its authors say the new data give a better idea of what’s going on in the classroom and how to improve it.
“You’re never going to be done with teacher evaluation,” Jiang says. “This whole process is really an evolution.”
But the CTU’s Caref has her doubts. She says it’s hard to separate the impact of REACH from the many other changes that have happened in the district in recent years — from a new state standardized assessment to CEO turnover — and “the fact schools are so underfunded and under-resourced.”
“How can we fix REACH so it would do a better job? I’m not sure it’s possible,” she says. “We would have to fix the school system.”