Nationwide testing of pre-kindergarteners is intended to gauge how well Head Start agencies are preparing youngsters to begin school, but the tool currently being used is far from being ready for such a high-stakes purpose, according to a recent Government Accountability Office (GAO) report.
Developed two years ago, the National Reporting System (NRS) aims to measure literacy, language and early math skills of 4- and 5-year-old children who are headed into kindergarten. It is administered in English and Spanish.
Federal officials also expect to use the test results to improve individual programs and target training and technical assistance. But a comprehensive study of the testing initiative by GAO—the federal agency charged with evaluating federal programs, policy and spending—noted several shortcomings of the test that would preclude it from being used to achieve these goals. Those limitations include:
Unclear how results will be used According to the report, which was released in May, the Head Start Bureau has not announced what level of progress it expects agencies to meet, how it will use test results to decide who needs training or how it will hold agencies accountable.
“As far as I know, they are not using this in any definitive way,” says Erikson Institute President Samuel Meisels, a critic of the NRS who also belongs to a team of experts who provided advice on test design and rollout. Twice a year the children take the test and twice a year the Head Start Bureau reports back to organizations that oversee Head Start programs, he explains.
Spanish version may not be reliable The report questions whether the Spanish version of the test produces reliable results. For one, the test may not adequately measure what Spanish-speaking children really know. The Spanish version has not been standardized to account for language differences among Spanish-speaking populations, and children from Puerto Rico, for instance, may recognize and use words that differ from those used by children from Mexico.
Also, the report notes that it is unclear whether results on the English version are comparable to those in Spanish. The tests are scored differently; some English answers are acceptable on the Spanish version, but not vice versa.
“Children may only know the Spanish name for something like knife,” says Leticia Martinez, a Head Start teacher at the Erie Neighborhood House site. “On the English test, they may pick up some English words, but not all of them.”
No one is monitoring curriculum and instruction In a survey of staff who had administered the test, GAO found 18 percent had changed what they were teaching children during the first year the NRS was given. Federal officials are not checking into whether such adjustments to instruction and curriculum put too much focus on developing cognitive skills at the expense of equally important social and emotional development.
“Teachers, administrators and parents are responsive to test scores,” says Shari Frost, the co-director of Literary Partners at National-Louis University. “If a preschool program has children with low scores, there will be pressure to improve those scores. Preschools will become more academic. The result will be the same thing that happened to kindergarten, and academic kindergartens have not improved literary rates.”
NRS has not been proven reliable or valid The federal assessment tests children twice a year in four areas: spoken English, vocabulary, letter recognition and early math. The 20-minute test is administered one-on-one by trained Head Start staff who use a script. However, experts who advised the Head Start Bureau during the test development warned that some pictures were confusing, some vocabulary words were inappropriate and asking children to name letters of the alphabet is not a valid way to measure how many letters children know.
“A ‘P’ doesn’t mean a thing to them at this age, unless the letter is in their name,” says Pamela Costakis, who oversees the state pre-kindergarten program at the Erie Neighborhood House. “They are concrete thinkers. You point to a ‘P’ and a child may say Paulo.”
Fine-tuning based on feedback
The study notes that the Head Start Bureau already has made some changes based on feedback from sites, and is considering the feasibility of other adjustments.
Spanish-speaking children, who must take both versions of the test, are given the more familiar Spanish version first to keep frustration to a minimum. Scripted test materials have been updated to help testers respond to a child’s behavior during the test. During the first year, when teachers were told to rigidly follow the script, they didn’t know what to do when children fidgeted, had to go to the bathroom or asked for a drink of water.
Under consideration for the future is testing a sample of children rather than every one. Centers have complained that the amount of time it takes to give the NRS takes away from other classroom activities. Peggy Riehl, director of Head Start programs at Boys and Girls Clubs in Chicago, documented how long it took to administer the test and document results for 149 children. “In the fall, it was 151 hours,” she says.
Also in the works: piloting a new section of the test that would assess a child’s social and emotional development.
In the meantime, Head Start Bureau officials say they will not use NRS data alone to measure program quality. Local organizations are following suit.
“We will use the data from the NRS as one of many variables; the test will not be used as a high stakes assessment,” says Mary Ellen Caron, commissioner of the Department of Children and Youth Services, which oversees some 16,500 Head Start slots in Chicago. “We will not put all our eggs in the NRS basket. We need many strategies.”