A New Look at Public and Private Schools: Student Background and Mathematics Achievement

The claim that private schools do a better job of educating their students than public schools -- an argument central to the push for market-style reforms -- is rarely questioned. But do the data back it up? The authors examined fourth- and eighth-grade NAEP mathematics achievement in over 1,000 public and private schools to arrive at a surprising answer.

By Sarah Theule Lubienski and Christopher Lubienski

A RECENT report of mathematics results from the National Assessment of Educational Progress (NAEP) highlighted this "major finding": "Public-school students scored lower on average than non-public-school students at both grades 4 and 8."1 Of course, this finding is nothing new. Indeed, it is part of the common wisdom in the United States that private school students outscore public school students on standardized tests. Furthermore, studies have suggested that this is true even when researchers account for the fact that the enrollment at public schools differs from the enrollment at private schools.

This belief is based, in part, on past studies involving the 1980 High School and Beyond dataset that found that private schools are more effective than public schools at boosting student achievement, including that of disadvantaged students.2 These studies of test performance, which controlled for some potentially confounding variables such as socioeconomic status (SES), affirmed widespread assumptions about the superiority of private schools. These assumptions, in turn, have influenced recent reform efforts promoting various forms of privatization of public schools, including the No Child Left Behind Act, which makes use of a variety of private sector sanctions for "failing" public schools.

However, the seminal studies on private school effects are becoming dated, with most based on a sample of students who began school over a generation ago. In addition, most of these studies were limited to high school.3 Some used rough measures of SES that may not have been sufficiently sensitive to the differences in populations served by public and private schools. Given the current interest in choice and privatization, it is an important time to examine current evidence relating to achievement in public and private schools.

While undertaking a broader study of mathematics instruction and equity, we became intrigued by an unexpected finding: when controlling for private school status and student background variables in our statistical models, we saw that mathematics achievement in public schools actually appeared higher than that in private schools.4 We decided to do a special substudy focusing specifically on achievement differences between public and private schools. Using a powerful SES variable created for the broader study, we were able to more carefully examine the question of whether the widely assumed "private school effect" is due more to the population of students served than to institutional effectiveness. The study focused solely on student achievement in mathematics -- a subject generally thought to be less influenced by family background and more influenced by institutional effects than other school subjects such as literacy.

The Role of Family Background

NAEP has been an important tool for monitoring trends in U.S. student achievement. It is the only nationally representative, ongoing assessment of U.S. academic achievement, measuring student performance at fourth, eighth, and 12th grades in mathematics and other subject areas. Our study used fourth- and eighth-grade data; eliminating 12th-grade data removed the issue of high school dropouts, which could bias our samples. Furthermore, the focus on students in the earlier grades reduces the likelihood that the students in the sample had transferred between public and private schools.

We analyzed achievement and survey data from the 2000 Main NAEP mathematics assessment,5 which we obtained from a restricted-use CD-ROM. (When the study was conducted in 2004, the 2000 assessment was the most recent for which raw data were available to researchers.) Fourth-grade samples included over 13,000 students in 607 schools (385 public schools and 222 private schools), and eighth-grade samples included over 15,000 students in 740 schools (383 public schools and 357 private schools). NAEP intentionally oversamples private schools in order to guarantee a sufficient number from which to draw statistical inferences.

In order to examine the extent to which SES differences between private and public schools account for the private school achievement advantage, we used a more comprehensive SES variable than the single variable -- eligibility for federally subsidized lunches -- that is typically employed. We began by creating a student-level SES variable. For grade 4, we combined the following six SES-related variables into one student-level SES variable:

• reading material in students' homes (newspapers, magazines, books, and encyclopedia);6
• computer access at home;
• Internet access at home;
• extent to which students' studies are discussed at home;
• school lunch eligibility; and
• Title I eligibility

With the exception of data on school lunch and Title I eligibility, which were taken from school records, this information was self-reported by students.

At grade 8, we combined these six variables with two additional variables: "mother's education level" and "father's education level," as reported by students.7 At both fourth and eighth grades, we combined the set of SES-related variables into a single SES variable using factor analysis (a statistical technique that creates new variables that best correlate with a set of variables).8 We then had a stronger, continuous variable that was more sensitive to SES differences than school lunch alone (which is restricted to just three levels -- free, reduced-price, and ineligible).

In the larger study of mathematics instruction and achievement, we used this student-level SES variable in several analyses. However, for the more focused analysis of public and private schools discussed here, we used only a school-level SES variable, which we created using a combination of our student-level SES variable and two school-reported variables: the percentage of students in the school qualifying for Title I funds and the percentage qualifying for subsidized lunches. These two variables were not continuous but instead contained only rough percentage categories (e.g., 1%-5%, 6%-10%, etc.). To create as strong a school SES variable as possible, we created a weighted average of these two school-reported variables with the richer SES information available regarding the students sampled from each school.9

We then used this new school SES variable to create four quartile groups: low SES, low-mid SES, mid-high SES, and high SES. We compared the mean mathematics achievement in public and private schools within each of these SES quartiles. In all of our analyses, we used the appropriate school weights and other NAEP-specific techniques that are necessary given the complex structure of NAEP data. Further details about the creation of SES variables and the larger study's methodology and findings are fully explained in a detailed report submitted to NCES.10

Summary of Findings

In our examination of overall mathematics achievement in public and private schools, our findings were consistent with previous studies. Specifically, we found that the mean mathematics achievement of private schools was significantly higher than that of public schools: 233 versus 227 points at grade 4 and 281 versus 274 at grade 8, for a difference of roughly .2 standard deviations at each grade level.

However, we also found, not surprisingly, that private schools enrolled larger concentrations of high-SES students. For example, at each grade level, while less than 40% of the public schools were of high SES (meaning that their SES was above the median for all schools sampled), over 80% of private schools were of high SES. So we wondered whether the achievement difference was attributable simply to SES differences or whether the private school advantage would persist within each SES group. In order to examine this issue, we compared the mean mathematics achievement of public schools and private schools within each of the four SES quartiles. The results are summarized in Figures 1 and 2.

As the figures show, within each SES quartile, the public school mean is actually higher than that of the corresponding private school mean at both grades 4 and 8. Specifically, public school fourth-grade means were 6 to 7 points higher than private school means within each SES quartile, and eighth-grade differences favoring public schools ranged from 1 to 9 points.

This situation is a classic case of Simpson's Paradox: although within each subgroup, public school means are higher than private school means, the overall private school means are higher than public school means because of the larger proportion of higher-SES students in private schools. These results call into question common assumptions about public and private school effects and highlight the importance of carefully considering SES differences when making comparisons of school achievement.

Concluding Thoughts

These results appear encouraging for public schools. Yet we offer two cautions for any analysis of NAEP data. First, most student background data are self-reported by students. Second, the achievement data are cross-sectional and thus not designed to measure growth over time or student movement between school sectors. Hence, correlations between school sector and achievement are not necessarily causal. In other words, one cannot conclude from this analysis that public schools are more effective at promoting student growth than private schools. There may be confounding variables not accounted for in this study that could help explain these patterns. On the other hand, our more complex models, which incorporated a number of additional variables, including race/ethnicity and disability status, consistently indicated that, when controlling for these confounding variables, public school means were higher than private schools means.

Our findings suggest that it is time for a critical reexamination of common assumptions regarding the effectiveness of public and private schools. As market-style reforms change the public school landscape, prompting many to call for various forms of privatization of schooling options, it is important to examine the evidence regarding whether private schools are, indeed, more effective than public schools. In our study, once we accounted for the fact that private schools tend to have higher-SES students than public schools, we actually found just the opposite of what was expected: public schools outperformed private schools within each SES quartile.

However, we should also note that, in our more complex statistical analyses of public and private schools, there were some ways in which outcomes appeared more equitable in private schools, a finding consistent with what Valerie Lee and Anthony Bryk found more than a decade ago. In particular, SES-related achievement gaps were smaller in private schools at grade 4, and the achievement disadvantage for Hispanic students was smaller in Catholic schools at grade 8. Yet, from the NAEP data, it is impossible to say whether schools actually narrowed those gaps or whether they simply attracted students with a narrower range of prior achievement.

Arguments promoting market-style reforms based on claims about the general superiority of private schools need to be reconsidered in light of the evidence this study provides. Further research is needed to illuminate the causes of differential achievement in public and private schools.

1. See http://nces.ed.gov/nationsreportcard/mathematics/results2003/schooltype.asp.
2. James Coleman, Thomas Hoffer, and Sally Kilgore, High School Achievement: Public, Catholic, and Private Schools Compared (New York: Basic Books, 1982); James Coleman and Thomas Hoffer, Public and Private High Schools: The Impact of Communities (New York: Basic Books, 1987); Anthony Bryk, Valerie Lee, and Peter Holland, Catholic Schools and the Common Good (Cambridge, Mass.: Harvard University Press, 1993); and John Chubb and Terry Moe, Politics, Markets, and America's Schools (Washington, D.C.: Brookings Institution, 1990).
3. This is the case with a more recent study as well. See Mikyong Minsun Kim and Margaret Placier, "Comparison of Academic Development in Catholic Versus Non-Catholic Private Secondary Schools," Education Policy Analysis Archives, February 2004, http://epaa.asu.edu/epaa/v12n5.
4. In this article we discuss one part of a larger study that used hierarchical linear modeling (HLM) to examine relationships among mathematics instruction, achievement, and equity. This study was funded by a NAEP Secondary Analysis Grant from the National Center for Education Statistics.
5. Two different NAEP assessments are administered to a nationally representative subset of students: the Long-Term Trend NAEP and the Main NAEP. The Long-Term Trend assessment was created in 1973 and has remained constant over time. In contrast, the Main NAEP is responsive to national curricular trends and currently assesses students' performance on both multiple-choice and open-ended questions over the five mathematics strands emphasized by the National Council of Teachers of Mathematics: number/operations, geometry, measurement, data analysis, and algebra/functions. There is also a third NAEP assessment -- State NAEP -- that is administered to samples from each state.
6. Students report whether they have each of these items in their homes, and the NAEP dataset combines the four responses into a single variable.
7. Because many fourth-graders have limited knowledge of their parents' schooling, NAEP no longer asks fourth-graders about their parents' education.
8. At each grade level, the factor analysis produced two factors with eigenvalues greater than 1, with one factor generally loading more heavily on Title I and lunch eligibility and the other loading more heavily on the remaining home environment variables. These two factors were then combined into a single SES composite variable. For more details, readers are invited to e-mail Sarah Theule Lubienski at stl@uiuc.edu.
9. Given that schools can vary in their definitions of who is and is not eligible for Title I, we considered lunch eligibility more reliable than Title I eligibility as a measure of school SES and weighted the variables accordingly.
10. This technical report, titled "Reform-Oriented Mathematics Instruction, Achievement, and Equity: Examinations of Race and SES in 2000 Main NAEP Data," is available upon request from Sarah Theule Lubienski at stl@uiuc.edu.