Beyond the Smoke and Mirrors: A Critique of the National Reading Panel Report on Phonics Ms. Garan hopes that readers will use her analysis as a starting point; take a long, hard look at the National Reading Panel's Teaching Children to Read; and draw their own conclusions -- without prejudice -- based on the research alone. By Elaine M. Garan

WHEN IT was released in April 1999, the report of the National Reading Panel (NRP) on the research on phonics instruction splashed down in the midst of a wave of controversy. In the ensuing philosophical turmoil, I reread with interest the "Editor's Page" in the March 1989 issue of the Kappan. At that time, many readers vehemently objected to the publication of "Debunking the Great Phonics Myth," Marie Carbo's critique of the research base for Jeanne Chall's Learning to Read: The Great Debate. I found it fascinating that the respondents tended to react to Carbo's challenge in ways that were in line with their basic philosophies rather than to critique the merits of her analysis. For many readers, the central issue was whether they agreed or disagreed with Chall's findings, rather than whether or not those findings were based on sound research protocols.

Indeed, some readers believed that it was inappropriate for Carbo to challenge Chall's research at all. They viewed the critique as a personal attack on Chall herself, and chastised the Kappan for publishing the article. Editor Pauline Gough responded, "It is entirely proper -- indeed, essential -- for a profession to examine its knowledge base at periodic intervals. Without prejudice. . . . We expect physicians to thoughtfully examine their knowledge base -- not to rely on intuition -- and then to modify their practices accordingly. Should we expect less from educators? . . . Knowledge advances through just such thoughtful give-and-take."1

I remind readers of this advice at the outset. Please try to set aside your own beliefs and "thoughtfully and without prejudice" critique with me the research summarized in the NRP's subgroup report on phonics. My purpose here is not to engage in the never-ending debate over whether code-first (i.e., phonics) or meaning-first instruction works best in teaching reading, the perennial issue that bubbles at the white-hot center of the interminable Reading Wars. Nor do I wish to take sides in the ancillary debate over the relative merits or demerits of quantitative versus qualitative methods of research. Rather, I request simply that we approach our mission as responsible professionals and ask only, "Is the NRP report on phonics based on sound research?" The following critique answers this question.

Background of the National Reading Panel

In 1997 Congress asked the director of the National Institute of Child Health and Human Development (NICHD) to appoint a panel to conduct a comprehensive investigation of research in the field of reading. The NRP was charged with compiling a report assessing "the status of research-based knowledge, including the effectiveness of various approaches to teaching children to read," and, if appropriate, with designing "a strategy for rapidly disseminating this information to facilitate effective instruction in the school." In its 13 April 2000 press release, NRP members and Duane Alexander, the director of NICHD, hailed the NRP report as a landmark contribution to education that "clearly articulates the most comprehensive review of existing reading research to be undertaken in American education." Donald Langenberg, chair of the panel, states that the NRP "was committed to identifying the most reliable research so it can be put into practice in all classrooms in America."

The NRP report certainly seems destined to have considerable clout. Backed by Congress and the NICHD, it has an unprecedented potential for affecting reading instruction in the U.S. Before the recommendations of the NRP report are put into practice in classrooms, it is vital, then, that we carefully examine its analysis from a research perspective and resist the temptation to react to the findings alone.

The Organization of the Report

Whenever research is presented, the citations must be clear and the references identified. However, readers of the NRP report will find the cross-checking of information at best confusing because the NRP published at least two different versions of the report, without distinguishing between the various editions. The original, posted on the NRP website in April 2000, did not include a complete table of contents. Not listed was the "minority report" by panel member Joanne Yatvin, which was originally included at the end of the nearly 600-page document. Then, without indicating that any changes had been made in the document, the panel at some later date posted a second version of the NRP report in which the text was altered and in which Yatvin's minority report was listed in the table of contents. Therefore, it is difficult for readers using different versions of the NRP report to check the information. (In this article, I will cite page numbers from the version currently posted on the Web.)

To further the confusion, the NRP report exists in three separate "formats," all of which have the title Report of the National Reading Panel: Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction. I have distinguished between the three formats (not to be confused with the different versions) as follows: 1) Summary indicates a 34-page overview, which appears when you go to the NRP website, www.nichd.nih.gov/publications/nrp/smallbook.htm; 2) Reports of the Subgroups refers to the nearly 600-page document that contains the data, discussion, tables, and appendices and can be downloaded as a PDF file from the same site; and 3) NRP video refers to a publicity video distributed by NICHD that promotes the research without any mention of the limitations of the panel's analysis.

Yet more confusion results from the fact that the panel incorrectly listed the studies used in the meta-analysis. Appendix A of the Phonics Subgroup Report,2 titled "Studies Included in the Meta-analysis" (Reports of the Subgroups, p. 2-145), includes a total of 49 studies. However, the discussion section and Appendix G of the Phonics Subgroup Report list only 38 studies. Such unfortunate inaccuracy is at odds with the panel's claims of scientific rigor and makes it difficult to see how "anyone can check the veracity of any claim."3 (Both Appendix A and Appendix G can be found at www.nichd.nih.gov/publications/nrp/smallbook.htm.)

Methodology Employed

Before examining the quality of the panel's analysis, let me present a brief overview of the process of selecting studies and of the methodology used to evaluate them. The process began with a series of questions concerning phonics instruction that the panel selected as being relevant; next the panel set criteria for inclusion.4 The NRP Summary states:

In what may be its most important action, the Panel then developed a set of rigorous research methodological standards. . . . This screening process identified a final set of experimental or quasi-experimental research studies that were then subjected to detailed analysis. The evidence-based methodological standards adopted by the Panel are essentially those normally used in research studies of the efficacy of interventions in psychological and medical research. These include behaviorally based interventions, medications, or medical procedures proposed for use in the fostering of robust health and psychological development and the prevention or treatment of disease. (Summary, p. 8)

In addition to requiring experimental or quasi-experimental control groups, other basic selection criteria were established. Each study had to "measure reading as an outcome" and include descriptions of participants, interventions, study methods, and outcome measures in sufficient detail to "contribute to the validity of any conclusions drawn" (Summary, p. 28). From the "potentially relevant" list of 75 studies, 37 were eliminated. The Summary continues, "Unfortunately, only a small fraction of the total reading research literature [38 studies] met the Panel's standards for use in the topic analysis." Nevertheless, the NRP conducted a meta-analysis on the unfortunately small number they eventually selected. As most Kappan readers know, a meta-analysis compares results across many research studies, all of which must assess a common outcome or dependent variable. Therefore, the greater the number of studies, the more reliable the results of the meta-analysis. The NRP's claim to a comprehensive, scientific review of reading research on phonics -- based on a mere 38 studies for a meta-analysis -- makes the reliability of its conclusions questionable from the outset.

The NRP determined that the studies it did include had to address the impact of phonics instruction on one or more of six isolated reading skills. The panel did not include reading comprehension or the application of phonics skills in authentic literacy events as necessary criteria in establishing what it termed a "general literacy" outcome. The discrete, isolated subcategories that the panel considered as representing "growth in reading" were:

• Decoding. This subcategory assessed the reading of words with regular spelling patterns in isolation -- for example, "cat," "fat," "mat."
• Pseudowords. This subcategory assessed children's ability to "read" pseudowords with regular spelling patterns -- for example, "dat," "wat," "gat."
• Word identification. This subcategory assessed children's ability to read words in isolation; some consisted of regular spelling patterns, while others did not.
• Spelling. This subcategory assessed children's ability to spell words. With children in kindergarten and first grade, invented spelling was accepted -- for example, "fet" for "feet" and "kr" for "car"; for older children (grades 2 through 6), only conventional spelling was accepted.
• Oral reading. No description of oral reading was provided.
• Comprehension. For younger children, this usually involved "extremely short (usually one sentence) 'passages.'" Often the assessments used favored "phonetically regular words" (Reports of the Subgroups, p. 2-115). For older children, the assessments usually involved passages from standardized tests or were developed by the researchers for training purposes.
• General reading/general literacy. The effect sizes (the size of impact of a treatment) for all the subcategories above were combined for each of the 38 studies to produce a measure of what was termed "general literacy."5 The general literacy effect size for each study was averaged across all the studies and groups. Although a study could assess six subcategories, most looked at no more than two or three. Some studies tested only one subcategory. Few studies tested for all six, and the majority of the comparisons (76%) focused on single words or pseudowords, rather than on comprehension. According to Appendix G of the Phonics Subgroup Report (Reports of the Subgroups, p. 2-169 through p. 2-175), only two of the individual studies (numbers 55 and 72) actually assessed or used a term such as "general literacy" or "reading growth" as an outcome measure. Thus "general literacy" was not assessed in 36 of the 38 studies but was a term assigned by the NRP that could -- and sometimes did -- refer to a single outcome, such as "word identification" (study number 52). The averaged, overall effect size of these "general literacy" outcomes across the treatments was determined by the NRP to represent "reading growth."

It is important to note that the means across the studies were inconsistent in two ways: 1) in the number of discrete skills (subcategories the studies assessed) and 2) in the types of skills those comparisons represented. In other words, the "general literacy" effect size from a study that looked at one or two outcomes ( e.g., "decoding" or "word identification") influenced the overall effect size as much as a study that assessed more subcategories. Therefore, an outcome assessing only isolated skills had as much of an impact on the overall effect size as a study that also tested for "comprehension." I will discuss the implications of this decision below. (See Table 1 of this article for the overall results of the NRP's meta-analysis and for effect sizes for the individual subcategories.)

TABLE 1. Results of the National Reading Panel's Analysis of Phonics Instruction
Effect Sizes for the Subcategories
	Decoding Words with regular spelling patterns only	Pseudoword Reading nonwords with regular spelling patterns only	Word Identification Some words irregularly spelled	Spelling	Oral Reading	Comprehension	General Reading Effect size of all categories
At-risk kindergarten and first-grade students	.98 high	.67 moderate	.45 small	.67 moderate Accepted invented spellings	.23 small Based on two K and four first-grade studies	.51 moderate Based on one K and 10 first-grade studies	.56 (K) and .54 (first grade) moderate
Grades 2-6, disabled and low-achieving readers*	.49 small to moderate	.52 moderate	.33 small	Not significant	.24 small	Not significant	Not significant
*The database may have included too few studies of low-achieving readers (eight) to draw firm conclusions.

The effect sizes for each subcategory, as well as the general effect size, were measured in standard deviation units. The mean for each of the subcategories in each study was calculated and expressed as 0. Thus an effect size of 0 indicates that there was no difference between the results for the treatment and control groups. An effect size of 1.0 indicates that the treatment group mean was one standard deviation higher than the control group mean. The NRP decided that an effect size of .20 is small, an effect size of .50 is moderate, and an effect size of .80 is large. For a more detailed description of the statistical procedures the NRP used, see the section "Coding the Data" in the Summary (pp. 30-33).

A Critique of the Research

Let us now examine the quality of the research in terms of whether: 1) the research base was appropriate for a meta-analysis, 2) the findings were generalizable based on the studies' student populations and sampling sizes, 3) the results were reliable, 4) the NRP's research was valid (that is, the research methodology examined what the NRP determined was important), and 5) the NRP accurately represented the data in its conclusions.

Appropriateness of the Research Base

Meta-analysis, a term coined in 1976 by Gene Glass, is an approach that makes it possible to summarize the results of individual experiments across a large number of studies. While a meta-analysis does not require that the same instrument be used in each of the studies, it does require that the studies involved assess the same conceptual outcome or dependent variable.6

The NRP's meta-analysis fails to meet both of these basic criteria. First, the panel included only 38 studies, a number that even it calls "unfortunately" limited (Summary, p. 8). Furthermore, after its rigorous screening process and after it had already analyzed the data, the NRP concluded, "There may [have been] too few studies of low-achieving readers in the database (only eight) to draw firm conclusions," and the results may not "even [be] reliable" (Reports of the Subgroups, pp. 2-117, 2-94). Those studies that the NRP dismissed as being of questionable reliability were also those that produced nonsignificant results for "spelling," for "comprehension," and for the "general reading" outcomes for low-achieving readers in grades 2 through 6. Here, we can conclude, based on the panel's own determination, that only 30 studies in the meta-analysis maintained even the possibility of reliable, generalizable results. I'll examine the reliability of the eight dismissed studies, as well as of the other 30, when I consider reliability below.

The NRP's research fails to meet the requirement that the studies' outcomes -- the dependent variables across studies -- be conceptually consistent. As noted previously, most of the studies (76%) assessed isolated skills, while only 24% required that children apply their skills to actual text. Proficiency in isolated skills and the authentic application of those skills (comprehension) are not conceptually consistent outcomes. For example, sounding out the word ilk and understanding its meaning are separate concepts.

In fact, the NRP's own meta-analysis clearly establishes that it is possible for children to perform phonics tasks in isolation and yet not apply those skills to authentic literacy events, such as comprehension or spelling. That is, with the exception of a weak, statistically significant effect size of .24 for "oral reading," the NRP's analysis shows that systematic phonics resulted in significant outcomes only on those subcategories assessing phonetically regular words or words tested in isolation for grades 2 through 6. Phonics had no statistically significant impact on tasks requiring authentic application. "Spelling" and "comprehension" outcomes for these children were not statistically significant (effect sizes of .09 and .12 respectively). In fact, in referring to the "older children" (those in grades 2 through 6), the NRP states, "Phonics instruction appears to contribute only weakly, if at all, in helping . . . apply these skills to read text and to spell words" (Reports of the Subgroups, p. 2-116, emphasis added). This discrepancy firmly establishes the conceptual dissimilarity between proficiency in isolated skills and in those required for the more complex processes needed for comprehension. In fact, the NRP alludes to the conceptual inconsistency in the following statements: "The imbalance favoring single words is not surprising given that the focus of phonics instruction is on improving children's ability to read and spell words" (Reports of the Subgroups, p. 2-92) and "the purpose of this practice [phonics instruction] is centered on word recognition rather than on comprehending and thinking about the meaning of what is being read. This may be another reason why effect sizes on text comprehension were smaller than effect sizes on word reading" (Reports of the Subgroups, p. 2-123).

In summary, then, the NRP report fails to meet the criteria for a sound meta-analysis for two reasons: 1) the small number of studies seriously compromises the reliability of the results, and 2) the dependent variables of the meta-analysis are conceptually inconsistent.

Generalizability of Results

Let me now turn our attention to the generalizabilty of the reported results to student populations and classroom environments throughout the nation. I will also examine the chronological context during which the studies were conducted.

Student populations. The NRP states that its meta-analysis is based on "studies that examined the effectiveness of phonics programs with three types of problem readers: children in kindergarten or first grade who were at risk for developing reading problems; older . . . disabled readers; [and] low achieving readers" (Reports of the Subgroups, p. 2-90). The NRP concludes, many pages later, "There were insufficient data to draw any conclusions about the effects of phonics instruction with normally developing readers above first grade" (Reports of the Subgroups, p. 2-116). Compare the preceding quotation with this one:

Findings . . . regarding the effectiveness of systematic phonics instruction were derived from studies conducted in many classrooms with typical classroom teachers and typical American or English-speaking students from a variety of backgrounds. . . . Thus the results of the analysis are indicative of what can be accomplished when systematic phonics programs are implemented in today's classrooms. (Reports of the Subgroups, pp. 2-96, 2-135)

Thus this meta-analysis began with three groups of problem readers, and it ended by generalizing the results to typical children in today's classrooms.

The NRP's analysis included studies with such titles as "Can Dyslexia Be Treated? Treatment-Specific and Generalized Treatment Effects in Dyslexic Children's Response to Remediation."7 Such narrowly focused research does not include "typical" children, nor can the results be generalized to them. "Typical" U.S. classrooms include normally achieving students, high-achieving students, and students with limited proficiency in English -- three major categories of children that the NRP chose to exclude from its analysis. Furthermore, in some states, such as California, children with limited proficiency in English are the "typical" student population. The insufficient data on normally progressing readers, coupled with the exclusion of other important student populations, indicate that the NRP phonics results are not generalizable to typical classrooms.

Classroom Environments and Methods Used in the Studies

Classroom environments and social contexts are important factors influencing instruction. Jeanne Chall and Shirley Feldmann found that there is considerable variation in instruction even in classrooms whose teachers profess to use the same methods.8 These differences are confounded when we include foreign contexts, and, of the 38 studies that the NRP analyzed, nine were done in foreign countries (Reports of the Subgroups, p. 2-112). Although the foreign countries involved in the studies were English speaking, we cannot assume that the social and instructional environments were typical of those in the U.S. and could be generalized to U.S. classrooms.

The majority of the studies (28 of the 38 as indicated in Appendix D of the Phonics Subgroup Report) involved commercial phonics programs as the treatments. The NRP states that "some" of the phonics programs were scripted, thus largely eliminating the role of the teacher. Furthermore, still others required "a sophisticated understanding of spelling, structural linguistics, and word etymology" (Reports of the Subgroups, p. 2-135). Again, the use of scripted programs in some instances and the need for teacher training to implement other programs are conditions that do not typify most classrooms.

Time periods. Of the 38 studies analyzed, 10 were conducted more than a decade ago (Reports of the Subgroups, Appendix G, p. 2-169 through p. 2-175). This time element is important because in some studies basal readers were used in treatment groups, while in other studies from different eras they were used in control groups. Basal readers, as the NRP notes, have changed over the years and are "presumably more eclectic" than they were in earlier years (Reports of the Subgroups, p. 2-124). As the NRP notes, the chronological context of the study affects the validity of the analysis as well as its generalizability. The panel states:

Perhaps the reading instruction experienced by students in control groups included more phonics than the reading instruction received by control groups in earlier years. . . . Table 2 identifies the control groups used in studies in the corpus. Whereas some groups were true "no-phonics" controls, other groups received some phonics instruction. It may be that, instead of examining the difference between phonics instruction and no phonics instruction, a substantial number of studies [in the meta-analysis] actually compared more systematic phonics instruction to less phonics instruction. (Reports of the Subgroups, p. 2-124)

Recall that the NRP staked a claim to rigorous, scientific screening, which suggests that it knew the characteristics of the studies selected for analysis. Now ask why the NRP decided only after analyzing the data that the treatment and control groups might have been inappropriate and that the outcomes were based on conceptually incompatible dependent variables. If this was the case, how did they then generalize the results to "typical" populations in the Summary and NRP video?

Here, let me pause to revisit one of the basic issues in research: because everyone in a given population cannot be tested, researchers must select a sample. That sample must be a subset of the larger population to which the results will be generalized. Here is Neil Salkind on the matter of sampling: "Good sampling requires maximizing the degree to which this selected group represents the population. . . . For it is only if the results can be generalized from a sample to a greater population that the results of research have meaning beyond the limited setting in which they were originally obtained."2 In terms of location, inconsistencies between treatment and control groups, time frames of studies, and especially the narrowness of sampling of student population, the studies in the meta-analysis do not represent a subset of the larger student population and classroom environments to which the NRP generalizes its findings. In light of this background information, consider carefully the panel's conclusions, "Thus, the results of the analysis are indicative of what can be accomplished when explicit, systematic phonics programs are implemented in today's classrooms" (Summary, p. 13).

Reliability

Before I discuss the importance of reliability in a study, we should pause to refresh our memories about related statistical terminology. Statistical reliability can be defined as the degree to which the results of a given study are predictable or consistent. Are the findings unique, or are they truly indicative of what we can expect from a given assessment or treatment? That is, does a test measure the same behavior or performance each time it is used, and does it result in similar outcomes? While the term is generally applied to testing instruments, the NRP applies both reliability and validity (discussed below) to the process and conclusions of its meta-analysis.10

Recall that the NRP, after its "rigorous" screening procedures, determined that the outcomes for low-achieving readers were "inconclusive" because there were too few studies (eight) to ensure a reasonable expectation of reliability (Reports of the Subgroups, pp. 2-117, 2-94). However, the NRP did not question the reliability of the nine studies involving students with reading disabilities, studies that incidentally produced slightly higher results in "comprehension" and "general reading" outcomes than those for "low-achieving readers." Eight was too few. Nine was just fine.

Moreover, throughout the report, the panel drew conclusions and generalized from findings drawn from far fewer than eight studies.

• Based on only one comparison, the panel concludes that phonics affects the "comprehension" of kindergartners; the "comprehension" results for first grade are based on six studies and 10 comparisons (Reports of the Subgroups, p. 2-115; see also Appendix G, p. 2-169 through p. 2-175).
• Based on only two kindergarten and four first-grade comparisons (two and four studies respectively), the panel concludes that systematic phonics affects the "oral reading" of "younger children" (Reports of the Subgroups, p. 2-115; see also Appendix G, p. 2-169 through p. 2-175). Both kindergarten studies had effect sizes that were not significant (.13 and .15). Two of the four "oral reading" effect sizes for first grade were not significant (0 and .03). These non-significant effect sizes were influenced by the two effect sizes in "oral reading" that were significant (8.79 and 2.18). It is not clear whether the panel adjusted these two effect sizes that appear to be large outliers (Reports of the Subgroups, Appendix G, p. 2-169 through p. 2-175). Based on so few studies with such uneven results, it is unlikely that the outcome for oral reading is reliable.

The studies analyzed by the NRP are also limited in terms of the numbers of children involved. While it is certainly important to keep in mind that we are dealing with a total of only 38 studies, it is also important to consider that 38% of them included fewer than 40 students. Moreover, two studies involved just 20 children (Reports of the Subgroups, Appendix G, p. 2-169 through 2-175). Thus the comprehensive, landmark research of the National Reading Panel involves a limited number of studies of a narrow population involving small numbers of children. And yet these results were generalized to include normal student populations in typical classrooms.

If we ignore the contradictions inherent in the NRP's conclusions as they are reported in the Summary and accept the statements in the lengthy Reports of the Subgroups, we see that the NRP itself recognized that, in terms of reliability and generalizability, its report is an extremely flawed research effort.

Validity

Let us pause here to revisit the concept of "validity" and its relationship to "reliability." Reliability is a necessary, but not sufficient, condition for validity. That is, to be valid, a measure must first be reliable. In summarizing the analysis of the reliability and generalizability of the NRP report on phonics thus far, note that we have agreed with the panel's conclusions that there are too few studies above first grade and too few studies below first grade from which to draw conclusions. By applying basic principles of research, we have established that the results of the NRP analysis are not generalizable to typical student populations and classroom environments. Given these severe limitations, it is hardly necessary to continue. Nevertheless, I will now very briefly address the issue of validity. Do the outcomes of the NRP's meta-analysis actually reflect what they were intended to measure?

The results of the meta-analysis produced a "general literacy" or "general reading" outcome. The NRP's intention in determining a "general literacy" outcome and what it meant by using the term "reading growth" remain unclear. That is, the NRP does not articulate a cogent definition of the reading process in Teaching Children to Read. The panel does, however, state that phonics is but a tool, "a means to an end," the ultimate goal of which is to ensure that children "know how to apply this knowledge in their reading and writing" (Reports of the Subgroups, p. 2-96). If this is, indeed, the NRP's philosophy with regard to the role of phonics in instruction, then the report is fatally flawed. Only 24% of the comparisons examined the impact of phonics on reading texts, and this 24% included both oral reading and comprehension. Of course, as any teacher of reading knows, it is possible for a child to "call" words without comprehending them. Therefore, less than the 24% of the total studies actually looked at what the NRP itself declared to be the essential outcome of phonics instruction: comprehension and the meaningful application of phonics skills (Reports of the Subgroups, p. 2-111).

The NRP cautions, "Educators must keep the end in mind and ensure that children understand the purpose of learning letter sounds" (Summary, p. 14). Ironically, the NRP neglected to take its own advice and focused primarily on studies that had nothing to do with what it declared to be the ultimate goal of phonics instruction.

Accuracy of the Data Reported

At the outset, let us note some important observations about educational research. First, it seldom proves anything. As a rule, educational research raises more questions than it answers, and it is notoriously vulnerable to interpretation both by the researchers and by the audience, all of whom bring their own philosophies to the table. Second, the results of educational research involve a leap of faith of sorts for the reviewers of the findings. In other words, no one was present when the research was conducted except the researchers and the subjects. Therefore, we rely very heavily on the accuracy of the data as they are reported. Thus it is appropriate that we now examine the consistency of the conclusions and accuracy of the data.

In order to understand the discrepancies between the data, the translation of those data into the NRP's findings, and the conclusions drawn by the panel, we must briefly review the organization and distribution of Teaching Children to Read. Recall that the report is distributed in three separate formats, each under separate cover. The brief, 34-page reader-friendly Summary is available through the NRP's website and was distributed by the hundreds at the International Reading Association Conference in May 2000. In this Summary, the 84 pages of data and discussion in the phonics section of the Reports of the Subgroups are condensed to fewer than four pages. A free, user-friendly publicity video is also available from the NRP's website, as are the other sections of the NRP report. Neither the Summary nor the video makes reference to any of the limitations of the study. Indeed, in some instances, the conclusions in these short versions directly contradict the data and the NRP's own statements in the Reports of the Subgroups.

It is important to distinguish between the various formats of the NRP report because it is reasonable to assume that teachers, parents, and even busy university professors will be more likely to read the Summary or perhaps view the video and overlook the actual data as presented in the Reports of the Subgroups. Given the status of the members of the panel and given the NRP's a priori claim to scientific, objective analysis, most people -- myself included -- simply assumed that the data and the conclusions, while they might be subject to interpretation, would at least be accurately reported in all formats. However, I discovered that this was not the case.

It is interesting that while the Reports of the Subgroups includes many tables, it doesn't include a clear table illustrating the point of the entire meta-analysis: that is, the results. I found it impossible to track the findings without such a table, and so I constructed my own. Table 1 shows my effort to clarify the results. The following examples show discrepancies between the data and the conclusions as they are presented in the Reports of the Subgroups and as they are publicized in the Summary and NRP video.

• Claim. "The meta-analysis revealed that systematic phonics instruction produces significant benefits for students in kindergarten through sixth grade and for children having difficulty learning to read" (Summary, p. 13).
• Contradiction. "There were insufficient data to draw any conclusions about the effects of phonics instruction with normally developing readers above first grade" (Reports of the Subgroups, p. 2-116). This statement directly contradicts the claim in the Summary and on the video that the results apply to the general student population. Furthermore, the effect sizes for children in grades 2 through 6 were not statistically significant for "spelling" (.09), "comprehension" (.12), or "reading growth" (Reports of the Subgroups, p. 2-116).
• Claim. "Across all grade levels, systematic phonics instruction improved the ability of good readers to spell. The impact was strongest for kindergartners and decreased in later grades. For poor readers, the impact of phonics instruction was small" (Summary, p. 13).
• Contradiction. "The effect size [for spelling for children in second through sixth grade] was not statistically different from zero. . . . [Phonics was] not more effective than other forms of instruction in producing growth in spelling" (Reports of the Subgroups, p. 2-116). There are no data, at any grade level, indicating that phonics helped children to spell conventionally. The studies for younger children accepted invented spellings, and the results for all children in higher grades showed no significant effect sizes. In addition, there were no results reported anywhere for "good readers" -- a population mentioned for the only time in the Summary.
• Claim. The NRP's results indicate "what can be accomplished when explicit, systematic phonics programs are implemented in today's classrooms" (Summary, p. 13).
• Contradiction. "There were insufficient data to draw any conclusions about the effects of phonics instruction with normally developing readers above first grade" (Reports of the Subgroups, p. 2-116).

While the results of any research are subject to interpretation, it is hard to reconcile the direct contradictions between the data presented in the Reports of the Subgroups and the conclusions stated in the Summary with the NRP's claim of scientific rigor and objectivity.

Summary

If Teaching Children to Read were a typical research study, published in an education journal and destined to be read only by other researchers, then I could simply end my analysis by saying that the panel's own words have established that the research base in its report on phonics is so flawed that the results do not even matter. However, as we have seen, this study has clout. It has a public relations machine behind it that has already promulgated the results throughout a very wide, very public arena as representing unbiased scientific "truth." Both Congress and the National Institute of Child Health and Human Development are committed to ensuring that the findings of the NRP report affect classroom instruction.

The conclusions of this study as reported in the Summary have generated headlines not only in education publications, such as Education Week and Reading Today, but also in such newspapers as USA Today, the Cleveland Plain Dealer, and the Indianapolis Star. It is, perhaps, too late to mitigate the effects of this widely distributed, widely publicized project. However, I can hope that this analysis will provide a tool for others who will want to delve more deeply into the findings of the NRP report before accepting or rejecting it on the basis of the philosophical hot topics that the research addresses. If our instructional methods are to be dictated by research, then shouldn't that research be sound? Let me close as I began, by reminding readers that "it is entirely proper -- indeed, essential -- for a profession to examine its knowledge base at periodic intervals. Without prejudice. . . . Knowledge advances through just such thoughtful give-and-take."11

I hope that you will use this analysis as a starting point; take a long, hard look at the NRP report; and draw your own conclusions -- without prejudice -- based on the research alone.

2. For ease of citation, we will use the title "Phonics Subgroup Report" to refer to Chapter 2, Part II, of the Reports of the Subgroups.

3. Timothy Shanahan, "National Reading Panel Report: Work Praised, but Distortion Fears Persist," Reading Today, June/July 2000, p. 4.

4. Reports of the Subgroups, pp. 2-89 through 2-90. The questions the panel sought to answer through its meta-analysis were:

• Does systematic phonics instruction help children learn to read better than nonsystematic phonics instruction or instruction teaching no phonics?
• Are some types of phonics instruction more effective than others? Are some phonics programs more effective than others?
• Is phonics instruction more effective when students are taught individually, in small groups, or in whole classes?
• Is phonics instruction more effective when it is introduced in kindergarten or first grade to students not yet reading or in later grades after students have begun to read?
• Is phonics instruction beneficial for children who are having difficulty learning to read? Is it effective in preventing reading failure among children who are at risk for developing reading problems in the future? Is it effective in remediating reading difficulties among children who have not made normal progress in learning to read?
• Does phonics instruction improve children's ability to read and comprehend texts as well as their decoding and word-reading skills?
• Does phonics instruction have an impact on children's growth in spelling?
• Is phonics instruction effective with children at different socioeconomic levels?
• Does the type of instruction given to control groups as part of a study to evaluate phonics make a difference?
• If phonics instruction is found to be more effective than less-phonics instruction, were the experiments that showed these effects well designed or poorly designed?

5. In most cases, effect sizes are calculated by subtracting the mean of the comparison group from the mean of the experimental group and dividing this number by the pooled standard deviation. Occasionally, other procedures are used. All calculations reported by the NRP were conducted with DSTAT, a software package for statistical analysis published by Erlbaum.

6. Neil J. Salkind, Exploring Research (Upper Saddle Creek, N.J.: McGraw-Hill, 2000), p. 181.

7. See study number 32, Appendix G: Maureen Lovett et al., "Can Dyslexia Be Treated? Treatment-Specific and Generalized Treatment Effects in Dyslexic Children's Response to Remediation," Brain and Language, vol. 37, 1989, pp. 90-121.

8. Jeanne Chall and Shirley Feldmann, "First-Grade Reading: An Analysis of the Interactions of Professed Methods, Teacher Implementation, and Child Background," Reading Teacher, vol. 19, 1966, pp. 569-75.

9. Salkind, p. 86.

10. "National Reading Panel Releases Report on Research-Based Approaches to Reading Instruction: Expert Panel Offers Its Groundbreaking Findings to Congress and the Nation," press release, 13 April 2000. The term "reliability" is used to refer to the general findings of the NRP report (Reports of the Subgroups, p. 2-94). The term "validity" is applied to the conclusions of the meta-analysis (Summary, p. 28).

11. Gough, p. 498.