Test Scores of Nations and States
By Gerald W. Bracey
Illustration © 1998 by Mario Noche
|
||
Test Scores of Nations and StatesBy Gerald W. Bracey Illustration © 1998 by Mario Noche | ||
OFTEN I wrap up speeches by projecting two overheads that show in microcosm what I think is the state of education. These overheads show eighth-grade mathematics data from the 1992 Second International Assessment of Educational Progress (IAEP-2) and National Assessment of Educational Progress (NAEP).1 The data look like this:
Top Scorers | |||
| Asian students, U.S. schools Taiwan Iowa Top third of U.S. schools Korea Advantaged urban students, U.S. Hungary2 White students, U.S. schools |
287 285 284 284 283 283 277 277 |
||
It is not clear why Asian American students score higher than anyone in the world, but at least part of the explanation no doubt has to do with the fact that, as a group, their parents are better educated and more prosperous than the nation as a whole (see Research, December 1997).
From this table we can see that, if Iowa seceded from the union, it would be the second-ranked nation in the world. The top one-third of American schools posted scores as high as Taiwan and Korea -- and in math, where we are such putative dolts.
Asian and white students in the U.S. make up about 70% of all K-12 students. Thus 70% of American students scored as high as or higher than the third-ranked nation.
At the bottom, things look quite different:
| Jordan Mississippi Hispanic students, U.S. schools Bottom third of U.S. schools Disadvantaged urban students Black students, U.S. schools |
246 246 245 240 239 236 |
These two tables show that the variability among the states is as great as the variability among countries, at least in this comparison. However, these tables do not show variability within states and nations. To see that, you need to go to a National Center for Education Statistics publication, Education in States and Nations (page 55 of the 1993 edition; page 154 of the 1996 edition). What you will find there is that the variability among states and nations is tiny in comparison to the variability within states and nations. The distance from last-ranked Mississippi to top-ranked Taiwan is 39 points, about 80% of a standard deviation on the NAEP scale. The distance from the bottom of Taiwan's distribution to the top is 150 points, roughly three standard deviations.
The variability of scores for Taiwan is particularly intriguing: the average Taiwanese students scored one point higher than the average Iowa eighth-graders. But Taiwan's 95th percentile is substantially higher than Iowa's. The best students in Taiwan outperform the best in Iowa. But Taiwan's fifth percentile is also substantially lower than Iowa's. The lowest-performing students in Taiwan score much worse than the lowest-performing students in Iowa.
I once had occasion to ask a professor in Taipei why Taiwan would exhibit such extraordinary variability. He said that the answer was quite simple: Taiwanese teachers give able students a great deal of attention and ignore the less able. I cannot verify this contention from firsthand experience, but it would certainly produce the kind of score distribution exhibited by the Taiwanese.
The link between NAEP and TIMSS (Third International Mathematics and Science Study) has been made in a somewhat different way. Eugene Johnson of the Educational Testing Service and Adriane Siegendorf of Education Statistics Services Institute have conducted studies linking the two. Their work was quietly released by the U.S. Department of Education in July 1998 (Report No. NCES 98-500).
For each state that took part in the 1996 NAEP mathematics state-by-state assessment, the researchers made an estimate about what the eighth-graders in that state would have scored had they taken the TIMSS tests. In earlier TIMSS releases, the 41 nations were grouped in three categories. The new report also used three categories: those that scored significantly higher than a state's estimate, those that scored significantly lower, and those whose scores did not differ significantly from the state's estimate.
By using these categories, the U.S. Department of Education hopes to discourage the "horse race" obsession that has plagued earlier international comparisons. This is a laudatory goal, but it also hides what I think is a significant outcome: that the scores of most countries are very close together. At the top are the four Asian nations, and at the bottom are about six developing nations. In between are a bunch of countries that look very much alike. Recall from the "Accuracy as a Frill Award" in the Seventh Bracey Report (October 1997) that, in the eighth-grade science assessment, Bulgaria outscored the U.S. in the percentage of correct responses (62% to 58%) and was ranked fifth among the 41 nations. Icelandic students got just 52% correct and landed in 30th place. A 10% difference separated Bulgaria and Iceland, but it yielded a difference of fully 25 ranks.
There are times when small differences have large consequences. In the most recent winter Olympics, in at least two events, the difference between gold and silver was one one-hundredth (.01) of a second. I don't think such considerations apply to as coarse a measure as a paper-and-pencil test.
Of the 40 states that participated in the state-level NAEP mathematics assessment, Iowa, Maine, Minnesota, Montana, Nebraska, North Dakota, and Wisconsin form the first rank. Each of these states had an estimated eighth-grade TIMSS score that is significantly exceeded by just six countries: Flemish-speaking Belgium, Czech Republic, Hong Kong, Japan, Korea, and Singapore. These six U.S. states scored significantly higher than 16 to 18 nations (depending on the state involved), with the remaining nations not significantly different.
A second wave of states -- Alaska, Connecticut, Massachusetts, and Vermont -- trailed the above-named six nations plus Slovenia and the Slovak Republic. Recall that the U.S. average of 53% correct trailed the international average of all 41 nations by two percentage points.
In science, the first-rank math group of six states was bested only by Singapore. This was also true of Colorado, Connecticut, Oregon, Vermont, Wyoming, and the Department of Defense Dependents Schools (DoDDS). Depending on the state, from 18 (Oregon) to 32 (Maine) nations scored significantly lower.
In IAEP-2, disadvantaged urban students scored lower than the lowest nation. The only city tested as a separate unit in TIMSS was the District of Columbia,3 and the nation's capital managed to outscore only South Africa, posting a score that was no different from those of Colombia and Kuwait. All 38 other nations scored significantly higher. Among states, Mississippi's performance was not much better than the District's; its score ranked higher than the three countries just named and no different from that of Iran. But Mississippi was significantly lower than the other 37 nations. Results for Alabama, Arkansas, Louisiana, South Carolina, and Tennessee were only marginally better than for Mississippi.
Incidentally, Iran's low estate in the TIMSS standings gives us an idea of how education is parceled out in that country. In the latest International Mathematics Olympiad, the U.S. team finished fourth; the Iranian contingent placed third.
How accurate are these estimates? The answer appears to be "quite accurate." There are actual TIMSS data for the state of Minnesota, as well as data estimated from NAEP. In comparing the estimated and actual mathematics scores for Minnesota, the researchers found that the same six countries bested Minnesota's eighth-graders. From the estimated score, 17 countries scored lower than Minnesota; from the actual scores, 16 did. England managed to edge into the "not significantly different" category on the actual test. Similarly, only Singapore was ahead of both the estimated and actual scores of Minnesota in science. Twenty-six nations scored lower than Minnesota's estimated science score; 28 scored lower than the state's actual score.
Although not showing variability in the same way as IAEP-2, this report does look at how many of the students in each state would score as well as the top 10% and the top 50% of the TIMSS sample. That is, if we dumped the scores of all students in all 41 countries together and then found where the top 10% (90th percentile) and top 50% (50th percentile) scored, how would students in the individual states stack up? In math the numbers look like this for the top states and for Mississippi and D.C. (numbers in parentheses are the upper and lower bounds of the precise estimated percentage):
| Math | Top 10% |
Top 50% |
||
| Iowa Maine Washington, D.C. Minn. Miss. Mont. Neb. N.D. Wis. |
4% (1-7) 6%(2-9) 1% (0-2) 7% (3-11) 1% (0-1) 6% (2-9) 5% (2-9) 5% (2-8) 6% (2-9) |
63% (54-72) |
In science, the top states and Mississippi and D.C. placed as follows. (Note that both the math and science tables use the actual data from Minnesota.)
| Science | Top 10% |
Top 50% |
||
| Colo. Conn. DoDDS Washington, D.C. Maine Mass. Minn. Miss. Mont. Neb. N.D. Ore. Utah Ver. Wis. Wyo. |
13% (8-18) |
65% (58-72) |
The math numbers for these states suggest that the dispersal of scores around the average is not as wide as in the 41 countries combined. Still, 7% of the students in Minnesota would score as high as the top 10% in all 41 countries together. In the top states, more than 50% of the students would score at or above the worldwide 50th percentile. We have a higher proportion of average and above-average students in our top states but a smaller proportion of stellar ones. Only 13% of the students in the District of Columbia would score as high in math as the average student in the 41 combined nations; only 17% in D.C. would score as high in science.
In science, the outcomes are much brighter. At both levels, the top 13 states plus DoDDS have percentages higher than those in the 41 combined countries. Three-fourths of the students in Maine and Montana would score as high as the average student in the 41 combined nations.
First world, third world, all right here at home. It makes no sense to speak of "American schools."
2. Strictly speaking, in IAEP-2 Russia and Switzerland were slightly ahead of Hungary at 279, but both countries failed to meet the criteria for representative samples in ways that would tend to elevate their scores. In TIMSS, all three countries were virtually identical: the Russian Federation, 501; Hungary, 502; and Switzerland, 506. The percentage correct for Hungary and Switzerland was the same, 62%. Switzerland did not meet the TIMSS exclusion rate criterion.
3. Public school students only. Some of the test statistics emanating from D.C., such as SAT averages, are misleading because they include private schools that account for almost as many D.C. students as the public schools. The College Board considers you a resident of whatever state you take the test in. This helps D.C. look good in press releases. It also aids New Hampshire, which is a small state with a number of residential college-preparatory schools. A New Yorker once told me that a number of her classmates went to North Carolina to take the SAT so that their state-level rank would be higher than it would be compared to test-takers in New York.
PDK Home | Site Map
Kappan Professional
Journal
Last updated 10 November 1998
URL: http://www.pdkintl.org/kappan/kres9811.htm
Copyright 1998 Phi
Delta Kappa International