PDK International | Phi Delta Kappan: Rotherham

Find more Kappan articles in the
searchable PDK Publications Archive

The Translators: The Media and School Choice Research

What the public knows about educational research comes primarily from the media. But, Mr. Rotherham points out, few reporters have the training to judge the quality or significance of studies, and the tendency is to emphasize controversy rather than solid findings.

By Andrew J. Rotherham

THE MEDIA play a pivotal role in determining how and why research influences public opinion with regard to policy. Political scientists Shanto Inyengar and Donald Kinder have shown through experimental research involving televised news how the presentation of news stories can have a powerful impact on what Americans think about issues.1 Prominent columns and articles, especially in the big East Coast papers, influence political behavior among the policy and political elites and offer signals about elite thought and opinion on key issues. The debates about the research on school choice illustrate the broader challenges the media face when translating research for public consumption.

At a superficial level, school choice is a relatively easy debate for the media to cover. It can be simplified into arguments for and against vouchers, charter schools, and altering the definition of "public" schooling, and these arguments are often boiled down to an easy framework of "public" versus "private." Likewise, the question of increases in test scores fits readily into a debate about whether school choice is "working" or not. While such framing greatly oversimplifies the issues, it nonetheless drives much of the coverage precisely because it offers easy contrasts.

Yet research usually offers nuance rather than stark contrasts, and the intersection between school choice research and journalism brings to the surface a key tension between social science and journalism more generally: their different tolerance for and approaches to handling "error" with regard to how definitive findings are. This is not to say that journalists are cavalier about error. On the contrary, most publications employ elaborate fact-checking and editing procedures. But, in addition to its reliance on formal, replicable methods of inquiry to answer questions, social science often parts ways with journalism in its approach to error.

There are two kinds of error in social science research: saying something is true when in fact it is false, or saying something is false when in fact it is true. The bias within social science is toward making the latter mistake, known more formally as a Type II error. In other words, when in doubt, favor the non-finding over the finding. Conversely, the natural bias in journalism is toward the Type I error, reaching the conclusion that something is true (publishing the story) even if it later turns out to be false. Put another way, while both fields prize accuracy, journalists are necessarily more concerned with the time-bound nature of news and events and so prize timeliness over certainty.

This is not a new story. Richard Colvin, executive director of the Hechinger Institute on Education and the Media, notes that this has always been a source of friction between social scientists and the press. But, he observes, it is more and more prevalent because of growing competition from online media and increasing pressure on news outlets to report news quickly.2

And it is a healthy tension. Newspaper stories are point-in-time projects, while the accretion of knowledge over time is the process in social science. People read newspapers to find out what is known at present. Research findings, which generally are part of a larger body of evidence and are often not definitive, must be presented in the appropriate context to be truly accurate and useful for readers. And, of course, single studies, regardless of their quality, should be considered cautiously.

The conflict arises when journalists seek a definitive angle to build a story around. Too often studies of test scores related to different school choice initiatives provide just such a slant. For instance, in a widely publicized episode, an analysis that offered no basis for causal claims, offered mixed results, and diverged from other research still landed on the front page of the New York Times in 2004 under the headline "Nation's Charter Schools Lagging Behind, U.S. Test Scores Reveal."3

In addition, despite their central role as translators and referees for the public, few reporters claim to really understand research methodology or feel competent to judge it. "Most journalists don't feel comfortable sorting out good research from bad research," says Colvin.4 Even Jay Mathews of the Washington Post, one of the nation's leading education writers, says that while he feels more confident about judging research than most of his peers, he still consults experts for their judgment.5 So those who most significantly influence the public debate about research are, by their own admission, poorly suited to adjudicate it.

Unfamiliarity can also lead to "on the one hand, on the other hand" accounts that leave readers to sort out multiple opinions.6 Grover "Russ" Whitehurst, director of the Institute of Education Sciences (IES) at the U.S. Department of Education, says, "Media reports of education research almost always try to create balance by quoting opposing points of view on the findings, as if the results were nothing more than opinion."7 Of course, some concerns about the validity and applicability of research are legitimate. But when given both sides of the issues, rather than some authoritative accounting, the public is understandably left confused -- or worse, misled.

In an effort to help reporters on short deadlines make better use of research, Whitehurst initiated the Rapid Research Response service at IES in 2003. At no charge, the service offers analysis of education studies within two business days. Whitehurst foresaw "a tool that would help the media understand the strengths and weaknesses of education research findings. This would help them avoid reporting weak research uncritically and might allow them to take an approach with strong research that was more in keeping with typical science reporting."8 In other words, Whitehurst hoped the service would lead the media to focus more on the relevance of the findings than the opinions or advocacy positions related to the study. IES has had not a single request for the service.9 The reasons are not clear, but the absence of requests raises important questions for those considering the rigor and reliability of media accounts of educational research.

Reporters and editors are understandably frustrated by the give-and-take of advocacy, which only confuses the issues more and makes journalists' jobs more challenging because knowledgeable, honest brokers are few and far between. In an effort to find seemingly reliable sources of research, reporters can inadvertently look to sources that lack explanatory power. For instance, many journalists point to government-funded studies comparing public and private schools and charter schools and other public schools as especially influential and newsworthy. Allison Mitchell, education editor at the New York Times, noted that the paper had covered studies comparing traditional public schools with public charter schools and private schools because the studies were government funded.10 Yet recent federally funded studies of student achievement and of public, private, and charter schools are descriptive, not causal. In other words, while they document the heterogeneous nature of broad classes of schooling, these studies can't tell us if these different kinds of schools help or hinder student learning. It is understandable that, in the adversarial world of school choice, journalists writing about research would seek a benchmark like public funding as a signal of the unbiased nature of a study, but it is still necessary for them to exercise some care and not inappropriately prioritize some studies over others.

Perspective also matters. Most education writers approach the subject from the point of view of local schools. Says the Washington Post's Mathews, "I'm a classroom reporter, not a policy reporter." Mathews is more interested in "using the research to identify what models are succeeding, which ones are not" than adjudicating disputes about the overall contours of choice schemes.11 Colvin agrees and notes that most journalists approach research seeking answers to the question "Is it working?" The problem, he says, is that such a question is inappropriate when applied to broad categories of schooling or educational inputs with substantial variation.12

However, the New York Times, the nation's most influential newspaper and a key source of information about education, does focus on education through a policy and political lens. Colvin describes the Times as covering "education from a political perspective, not from a research perspective, in terms of what we know or how our knowledge has evolved." This orientation, he says, can at times be an "awkward imposition of a frame on a story."13

Perhaps the most noteworthy example of this political perspective was the Times' front-page story on charter school student achievement, which I mentioned above.14 In August 2004 the Times published a story about the performance of charter public schools and traditional public schools on the National Assessment of Educational Progress (NAEP), a nationally representative test of student achievement.15 Several years earlier, charter school supporters had sought to have charters included in the NAEP sample, and the first charter data from NAEP became available in 2004.

The account in the Times was based on the release of data by the American Federation of Teachers (AFT).16 The data had been made publicly available on a government website, but the U.S. Department of Education had not yet released any formal analysis. The AFT should have presented the data in a way that made the findings more understandable to those unfamiliar with statistics, and it should have made the necessary caveats more obvious. The AFT's charge, repeated by the Times, that the Bush Administration was seeking to squelch the data (despite its public availability) added drama to the story. Although the AFT did nothing to tamp down the ensuing firestorm, it did not actively misrepresent the data, and the burden of skepticism should fall harder on the Times anyway. After all, the AFT is an interest group doing what interest groups are supposed to do. It was the characterization of the data and the placement of the story by the Times, more than the underlying AFT report -- which probably would have garnered little attention without the coverage -- that set off the furor.

In particular, the Times story cast charter schools as a Bush Administration initiative, despite the bipartisan pedigree of the reform idea; included a chart that did not differentiate between findings that were statistically significant and those that were not; and failed to give readers context about what NAEP was, primarily the fact that it could not control for prior achievement of students, so that the effects ascribed to different kinds of schools might have nothing to do with the schools themselves. The article also ignored a substantial body of research from studies with more explanatory power about charter schools, studies that were at odds with the thrust of the story.17

This episode, which set off an ongoing debate, demonstrates the power of the media to frame a debate on policy, and it also shows how discrete pieces of research that do hit the public debate are often shorn of any sort of context. In a single episode, all the liabilities of the debate about school choice research were highlighted: a hyperadversarial advocacy climate, reporters who have trouble making sense of complicated research evidence, and the ensuing inappropriate use of data points.

Because school choice initiatives can radically change the power arrangements in education, it's naive to expect advocates on any side of the debate to suddenly become completely fair and balanced. And, at least to date, the evidence is mixed about various school choice schemes and often depends on the questions being asked. As a result, there is plenty of fodder for advocates on both sides.

But the traits that make school choice research so hard for journalists to cover are hardly unique to that debate. Consequently, education reporters have their work cut out for them. It used to be that covering education was pretty much a "he said, she said" sort of affair, and balance meant giving both sides their say. Today, as a greater emphasis on empiricism takes hold in debates on educational research and policy, it is vital that reporters present this work in a way that allows readers to determine what new information does or does not mean. That is not easy work and is usually not black or white, but it is essential to a vigorous and healthy political debate about schools.

Unfortunately, reporters and editors say professional development for reporters is a low priority at most media outlets. Like any organization, news organizations are constantly dealing with the challenge of too much work and too few people. In addition, because reporters, especially at regional and local papers, tend to move off the education beat fairly quickly, editors are understandably skeptical of the returns they will get from professional development. Still, the sort of training that the Hechinger Institute on Education and the Media offers is one of the best ways that reporters can learn the ins and outs of consuming research and writing about it. Short of that, services like the one that IES' Whitehurst offers can also provide valuable feedback. At the end of the day, while researchers can do more to make their work - and its limitations - easily accessible for readers, there is no substitute for the intelligent consumption of research. Those skills, however, can only come with training or the help of experts. As education thankfully becomes more empirical as a field and disputes over issues like school choice engage more research, those who write about education issues have to become more comfortable with empiricism, too.

1. Shanto Inyengar and Donald R. Kinder, News That Matters (Chicago: University of Chicago Press, 1987).

2. Interview with Richard Colvin, 12 March 2007.

3. Diana Jean Schemo, "Nation's Charter Schools Lagging Behind, U.S. Test Scores Reveal," New York Times, 17 August 2004.

4. Colvin, op. cit.

5. Interview with Jay Mathews, 11 March 2007.

6. See, for example, the New York Times coverage of the Krueger/Peterson debate: Sam Dillon, "Report Defends Vouchers but Fails to Quell Debate," New York Times, 13 June 2003.

7. E-mail correspondence with Grover Whitehurst.

8. Ibid.

9. Ibid.

10. Interview with Allison Mitchell, 15 March 2007.

11. Mathews, op. cit.

12. Colvin, op. cit.

13. Ibid.

14. Schemo, op. cit.

15. For more information on NAEP, see www.nationsreportcard.gov.

16. Charter School Achievement on the 2003 National Assessment of Educational Progress (Washington, D.C.: American Federation of Teachers, 2004).

17. See, for instance, Bryan C. Hassel and Michelle Godard Terrell, Charter School Achievement: What We Know, 3rd ed. (Washington, D.C.: National Alliance for Public Charter Schools, 2006).

ANDREW J. ROTHERHAM is co-founder and co-director of Education Sector, a national education policy think tank. He serves on the Virginia Board of Education and writes the blog Eduwonk.com.