|
||
TIMSS, Rhymes with 'Dims,' As in 'Witted'By Gerald W. Bracey Simplistic and misleading statements seriously distort what the TIMSS official report actually says, Mr. Bracey avers. For the real story, read on. |
AS IN THE movie The Official Story, the official story of the Third International Mathematics and Science Study (TIMSS) is not the real one. Indeed, for TIMSS, the official story is not even the same story contained in the official report. The simplistic and misleading statements of TIMSS co-director William Schmidt, National Center for Education Statistics (NCES) Commissioner Pascal Forgione, and U.S. Secretary of Education Richard Riley seriously distort what the TIMSS official report actually says. Moreover, I am glad that Bill Clinton already has a good job as President because his TIMSS-related comments on the impact of poverty on learning reveal a logic deficit that would prevent him from becoming a sound educational researcher. Clinton said that poverty couldn't be the cause of the low standing of U.S. 12th-graders, because our fourth-graders did so well. (Evidently, the President is not familiar with cumulative effects.)
The U.S. Department of Education states that the math and science literacy tests measure the knowledge of mathematics and science needed to function effectively in society as adults. For this, the department earns my "Unmitigated and Overweening Horsehockey Award." Readers, call the U.S. Department of Education and ask first for a definition of "function effectively in society." Then ask someone to tell you how the department knows that the TIMSS math and science tests measure this ability. The published sample items strike me as quite reasonable things to teach students, but to go on to say that this stuff is "needed to function effectively in society" is utter nonsense. (It recalls the similar nonsense in the 1970s over minimum competency testing and the debate then over what skills are "essential" to function in society.)
I know, I know, skeptics will say that I've gotten locked into this position as the nation's leading "defender" of public schools and that I am now trying to put a good spin on what everyone else agrees are "devastating results" (Schmidt's phrase) and "unacceptable" (the President's word). Well, then, check out the facts and decide for yourself. (If only reporters had done so!)
My take is that the physics and advanced mathematics scores are beyond redemption; no statistical manipulation can make them wholly credible. They are condemned to lie in a muck of uncertainty. If the math and science literacy scores were accurately calculated, factoring in appropriate variables, the United States would be about average. Not a cause for celebration, to be sure, but not the disaster so far painted.
NCES Commissioner Forgione said that the graduation gap between the United States and other nations had been closed, meaning that 12th-graders in various countries are now comparable. This is just not true. Mathematics and Science Achievement in the Final Year of Secondary School, the official TIMSS report, says, "There remains considerable variation among countries in completion rates" (p. 17). What the table on page 22 actually shows is that there remains considerable variation among countries in secondary school enrollments. When one considers that for most nations this is the proportion of students between the ages of 12 and 17 who are enrolled, the variation is surely much larger for graduation rates.
Here are a few of the other statistical knee-slappers. The TIMSS "kids" in Iceland are about the same age as American college seniors. That's right, folks, the Icelandic students who were compared to our 12th-graders averaged 21.2 years in age. Those in Germany, Norway, Italy, Austria, Sweden, and Switzerland were merely closer in age to our college sophomores. For this we paid $31 million? (Do you think that spending $31 million, while other nations ponied up a pittance, has anything to do with our making such a big deal over the results of the study?)
Norwegian kids bested the rest of the world in physics. The question is: Why didn't they do even better? After all, they had been studying physics for three years. Ditto the second-place Swedish kids. This ought to put to rest a theory abroad in the land: that U.S. students would have looked worse if any Asian nations (who topped the rest of the world at the eighth-grade level) had shown up for the 12th-grade testing. Unless they, too, had multi-year programs, they would no doubt have failed to equal the Scandinavian contingent.
Apparently, nobody saw fit to mention at the "official story" press conference that the American TIMSS cadre included our students who were taking precalculus. Is this important? Well, those students scored fully 100 points lower than students who had actually taken calculus, possibly because 23% of the questions presumed that test-takers had absorbed a calculus course. American students with calculus actually under their belts answered correctly only one item less than the international average of all 21 nations.
And how's this for a statistical fluke? On the island nation of Cyprus, students ranked 18th of 26 nations in the TIMSS grade-4 math assessment, 37th of 41 nations in the grade-8 math assessment, and 20th of 21 nations in grade-12 math literacy. The best Cypriot students did not score as high as the average students in some countries. The average math literacy score for the top 25% of students in the 21 nations combined was 585. America's best 25% scored 559, and the top quarter of Cypriots scored a mere 501. Yet the Cypriot students outscored us in "advanced mathematics" at the 12th-grade level. Is something screwy here or what? My guess is that Cyprus has a dual education system that coddles an elite and largely ignores the rest. Either that or they cheated.
In fact, the official TIMSS report says that students in lycea (as academic secondary schools in Cyprus are called) can pursue five courses of study, one of which is math and science. And lo, these are the students who were tested. Cyprus also has "technical schools," in which students pursue vocational studies or math and science, and those from the latter group head to universities. The math and science students in the technical schools were tested -- the vocational students were not. Altogether, these two groups still accounted for a smaller proportion of the student body than the Americans who were tested.
The U.S.-Cyprus difference in the structure of schooling is just one of many problems of differing cultures that afflict these studies and cannot be solved by taking ever more precise samples. Because of these differences, international comparisons on test scores can never be very meaningful. (What can be meaningful is to look at what other countries do in their schools and decide if it makes any culturally relevant sense for us to do it, too.)
Here's a more noteworthy example of cultural differences: fully 55% of American students reported themselves working at a job more than 15 hours a week.1 Research studies have reported that, for us, jobs and schoolwork exhibit a curvilinear relationship. Kids who work up to 15 hours a week do better in school than those who don't work and those who work longer hours. The TIMSS results corroborate this. American students who reported that they worked 10 hours a week actually scored above the international average. Those who worked more than 25 hours a week were 60 points lower. (In almost all other countries, very few students work at all.)
Who knows what other cultural and social factors affect these data? Such factors often come to light only by accident. For instance, in the First International Adult Literacy Survey (FIALS), the United States did pretty well overall. Only Sweden had a larger proportion performing at the highest two levels. But our immigrants in the study had severe reading limitations, no doubt because of having first learned a language other than English. On the other hand, FIALS also revealed that Canada has a higher proportion of immigrants who read at the highest two levels (28%) than of native-born Canadians (20%). What could explain this apparent absurdity? Turns out that Canada has an aggressive policy of recruiting highly skilled, highly educated immigrants to come live in the land of the maple leaf. Thus these scores that at first glance appear to say something about the quality of education are in fact strongly influenced by immigration policy.
Finally, the official story about TIMSS has not reported that most countries did not meet the TIMSS criteria for participation. To have their results counted, countries were supposed to have a combined student/school participation rate of at least 75% and a sample of students representative of at least 90% of all students eligible to participate. Only nine countries met the first criterion, only 14 met the second, and only five met both. Some countries excluded 22% of eligible students (Cyprus) or 30% (Italy) or even 43% (Russian Federation). Reporting the results for all 21 countries is thus comparing the incomparable, something that is strictly forbidden under normal rules of research.
I say again as I have said so often in the past: there are serious problems in American schools, and our poor rural and urban schools need the equivalent of a Marshall Plan. But we should attend to problems that actually exist. Those that were purportedly revealed in the official story of TIMSS do not. Forgione has called TIMSS "rough around the edges." I say rotten at the core. The official TIMSS story is an exercise in political rhetoric and comes very close to being a hoax perpetrated on the whole world.
PDK Home | Site Map
Kappan Professional
Journal
Last updated 8 May 1998
URL: http://www.pdkintl.org/kappan/kbra9805.htm
Copyright 1998 Phi
Delta Kappa International