 |
|
|
|
|
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
|
|
||||||||||||||||||||||||||
|
|
Find more Kappan articles in the The Project on High Performance Learning Communities: Applying the Land-Grant Model to School Reform The authors describe the evolution of the Project on High Performance Learning, established in 1989 in response to the needs of policy makers and practitioners for a more fully developed and practical knowledge base about "what works" in education reform. ALTHOUGH schools and practitioners are under enormous pressure to create successful change in complex organizations, they are typically offered few tools and little guidance about the processes for change or about how to translate the big recommendations "out there" into what reform actually needs to look like in the schools. In this article we describe the evolution of the Project on High Performance Learning Communities (hereafter referred to as "the Project"), which was created in response to the needs of policy makers and practitioners for a more fully developed and practical knowledge base about "what works" in education reform -- both at the school level and in the conditions and policies that surround schools and shape the potential for creating and sustaining effective reforms in the teaching and learning context. The Project was established in 1989 at the Center for Prevention Research and Development at the University of Illinois. It recently moved its primary home to the National Center on Public Education at the University of Rhode Island. The Project as a whole addresses the need for an evidentiary base to guide efforts to improve K-12 education. It conducts large-scale, theory-driven, and theory-testing evaluations of the impact of major reform initiatives. A primary goal is to enhance the development of a knowledge base for guiding systematic and increasingly effective reforms in policy and practice. For school reform efforts in the United States and elsewhere to move beyond the recruitment of "heroes" and "heroines" and the removal of "villains" to widespread systematic reform that can be replicated across the nation by all school systems, a sound research base is essential. The Project is built on a model that values collaboration between practitioners and researchers. The organizing metaphor for such a partnership was borrowed from the relationship between land-grant universities and the agricultural community. That is, the model for research and development allows for continuous redesign and refinement. As lessons are learned, they are introduced into participating schools and undergo further testing on an ongoing basis, both in individual schools and across school sites. Since these are real schools with students in them, it is not acceptable to either the researchers or the practitioners to withhold innovation or change to meet the requirements of a research design that stays static until the end of the "test." We therefore had to develop and employ a research design in which changes in schools were viewed not as "noise" that had to be dealt with but rather as opportunities to learn more about how change unfolds and the impact it has. Another aspect of this "land-grant approach" is that the community both defines the problem to be solved and tests the adequacy of the answer. It does no good to develop models for reform that cannot be mounted in typical schools and districts. All too often we have heard from researchers that the recommendations for reform would "work" if only we had better schools and systems. The Project is practical. If the schools we work with tell us that the elements of reform we identify as important cannot be mounted without creating conditions that are impossible for the average school to replicate, then, while what we learn may be interesting, it does not help us to address our core concern -- understanding the nature and development of public education that is highly effective for all students. The role of the university/research partners in the Project is to synthesize the most up-to-date, state-of-the-art information and transfer it to those directly involved in the reform process in order to enhance the capacity of schools and communities for effective change. It is also to develop strategies that allow for the systematic testing and accumulation of a general knowledge base and to provide this information to specific schools in ways that assist them in local improvement efforts. Similarly, researchers at land-grant universities developed broad lessons that took the nation from the dust bowl to a state of surplus while, at the same time, providing information to individual farmers that enabled them to manage their own crops more effectively. This model demands an evaluation strategy that provides reform initiatives with evidence about lessons learned across sites and helps schools to conduct self-reviews of their practices and policies in order to set local goals in light of those lessons. As we will describe later, the development of the High Performance Learning Communities Assessments (HiPLaCes-A) was guided by the need for an evaluation that allowed for local self-review and assessment as well as for understanding ongoing change and growth in the process of educational improvement at the cross-school level. One division of the Project focuses specifically on middle-grades reform. For those concerned with the education of adolescents, the calls for reform have been no less strident or diverse and the range of options proposed no less confusing than for K-12 education in general. Further, the reform of middle-grades education has come under the same scrutiny and attack as has other education reform in recent years, with the same signs of impatience and disillusionment emerging. Indeed, although many states and communities have moved toward implementing the recommendations of the National Middle School Association and related groups, some have judged the results so disappointing that they have decided to drop such efforts. More than 40 years ago Edward Olsen argued that the "schools of tomorrow" would combine a focus on high levels of academic attainment with a concern both for the development of the whole child and for that child's ability to assume a productive role in a democratic society. Between Olsen's prophetic comments and the publication of Turning Points: Preparing American Youth for the 21st Century, released by the Carnegie Council on Adolescent Development, several major works called attention to an "embarrassing lack of knowledge" about early adolescence in general and specifically to the serious "lack of fit" between the developmental imperatives of early adolescence and the schools established to "enhance the power of [adolescents'] intellect" and to socialize them to adulthood.1 Similarly, Turning Points noted that "a volatile mismatch exists between the organization and curriculum of middle grade schools and the intellectual, emotional, and interpersonal needs of young adolescents." This mismatch, the Council noted, impedes learning and preparation for adult life; it increases adolescents' levels of risk and their vulnerability to a wide array of socio/emotional problems and self-destructive behaviors. Turning Points called on the education sector "to start changing middle grade schools now," and it offered educators comprehensive and researchable constructs and exemplars of those constructs to undergird their reform efforts.2 A Developmental Framework For What Works A major objective of the Project on High Performance Learning Communities has been to develop a knowledge base regarding the nature of learning contexts that provide for the highest levels of performance, achievement, and positive developmental outcomes for children and youths. Moreover, we want to address these issues in ways that will answer the core questions of policy makers and practitioners. We have focused on four such questions:
It is important that we be quite clear at the outset about what we are to employ as a standard for judging "what works." For us, the bottom line is that public schools work to the extent to which they, in partnership with the communities they serve, enable students to learn and achieve at high levels and help them become healthy, responsible, and productive citizens in our democracy. If reform efforts do not produce gains in the achievement, learning, performance, and adjustment of students from all backgrounds, then no matter what else they produce, we would judge them to have fallen short of this standard. Our standard brings to light the fact that an implicit operating assumption of much of education reform is that there will be a certain number of "acceptable casualties." This assumption has been reflected in education funding, in state and local goals for K-12 education, and in discussions of the privatization of schools that accept that some schools will "fail" and market forces will then push them out of the system. Unfortunately, the students in schools that fail will have had substantial damage done to their education. The assumption of a certain number of acceptable educational "casualties" is no longer viable in today's world, because to be an educational casualty is now, more often than not, to be a social and economic casualty as well. Although in the past those who were educational casualties could still find well-paying jobs that allowed them to support families and participate fully in society, this is no longer true in today's economy. Insistence on "no acceptable casualties" as a core value of the Project helped to narrow the choice of models that we considered as the basis of our work. Thus we omitted reforms that do not explicitly seek to address the creation of learning environments that provide for equity of educational opportunity for all students, particularly those who are socially and economically disadvantaged. Similarly, we omitted reform initiatives that fail to recognize that the primary business of public education -- providing the metric against which it is judged -- is to enable all students to learn, achieve, and perform at world-class levels. The undeniable linkage between learning and achieving at high levels on the one hand and full participation in society and the economy on the other makes our definition of "what works" one that transcends liberal and conservative ideologies. Given this standard for "what works," let us now turn to the identification and development of the model for reform that might best lead to the achievement of these goals. A Guiding Framework for Reform The need for theory-driven evaluation research on school reform is made clear by Valerie Lee and Julia Smith, who state, "Despite its growing popularity among school people and educational policy specialists, the reform movement, embodied in the term 'restructuring,' rests on thin and inconsistent theory."3 Seymour Sarason notes that there is a critical need for a vision around which to organize school reform efforts if they are to have any hope for success. He states, "What I mean by a vision [is] a central idea, a big idea, that radiates out and magnetically attracts and connects other ideas."4 Reforms that are derived from a central "big idea" exhibit a coherence of elements and strategies that make sense to participants and whose steps follow logically. The whole is more than a loose knitting together of discrete components. Each element at each systemic level can be measured against the degree to which it supports or impedes the attainment of the central vision. The "big idea" that best summarizes the questions with which the Project is concerned is "How do we create educational contexts in which all children and youths are nurtured and challenged in ways that lead them to be highly effective learners and healthy, responsible, and productive citizens in our democracy?" The first step was to settle on a comprehensive developmental model for reform that would be adequate for understanding the key dimensions of educational environments and the interrelationships among them as they relate to students' developmental outcomes. We needed a theoretical framework that would guide all actions and against which all reform elements could be judged and evaluated. If the goal was continuous refinement and improvement of education reform, rather than a "thumbs-up, thumbs-down" test, then the model we employed needed to be one whose assumptions were clearly testable against the outcomes sought and one that provided both guidance and flexibility concerning the incorporation of lessons learned. Fortunately, there existed both a broader formulation of such a model and a comprehensive statement for the reform of the education of young adolescents that directly reflected this broader formulation. The overarching theoretical framework that guides our work has been described and developed by Urie Bronfenbrenner, Arnold Sameroff, and others and has been variously termed "developmental ecology" or the "transactional/ecological perspective."5 This theoretical framework provides a means for understanding the ways in which the contexts in which children and youths grow shape their development and, in turn, the ways in which those contexts are shaped by those children and youths. Adoption of this broader theoretical framework led us directly to Turning Points as the statement on reform that best emphasizes the need for high-quality learning environments that enhance development and seek high achievement for all students. It has the additional goals of producing healthy, responsible, and productive citizens. Consistent with a focus on changing the whoIe school as a developmental context for learning, Anthony Jackson, the primary author of Turning Points, notes that the report is "an attempt to offer a plan for action for transforming middle grade schools into learning environments suited to the needs of young adolescents and equal to the challenges of a rapidly changing world."6 Turning Points proposes eight primary recommendations for reform (see Figure 1) that make up an integrative and comprehensive framework for school change within the broader policy and resource context. These recommendations are highly congruent with those of a number of other reform initiatives -- both those directed at the middle grades and those that focus on other levels of K-12 education. There is no shortage of statements about the best ways to improve middle and secondary schools in the United States. Among the reform recommendations whose impact is most widespread are those of the Coalition of Essential Schools, those set forth in the National Middle School Association's This We Believe, and, most recently, those offered in Breaking Ranks, the report of the National Association of Secondary School Principals in partnership with the Carnegie Foundation for the Advancement of Teaching on the high school of the 21st century. A number of states have also developed recommendations for education initiatives that build on these national recommendations.
Thus it is clear that the broad recommendations in Turning Points and many of their sub-elements are neither new nor unique. Indeed, one may suggest that there is a more general set of "Common Recommendations for Developmentally Based Reform." Turning Points is unique, however, in the organization of its recommendations, in its consistent level of analysis, and in its presentation in a way that easily translates into a comprehensive, testable model. Turning Points does not present us with yet another set of recommendations that seem random, arbitrary, or only loosely connected. Rather, each recommendation is one that may be necessary, if not sufficient, for some or all of the other elements to be successfully implemented. Each recommendation is logically connected to and derives from the central vision of the report. Implementation and Evaluation Having determined that the framework provided by Turning Points afforded the highest potential for achieving the outcomes we sought, the next step was to track the actual implementation of the Turning Points recommendations and to evaluate their impact. Such evaluation would allow for continuous refinement and improvement of the model, which in turn would enable schools for young adolescents to become ever more effective in reaching their goals for achievement and development. Serving as partners in the Project's middle-grades reform work and providing the data for the Project's evaluations have been four initiatives dedicated to realizing the vision set forth in Turning Points: the Illinois Middle Grades Network, the Middle Grade School State Policy Initiative (MGSSPI), the Middle Grades Improvement Program, and the Middle Start Initiative. (See sidebars for descriptions of the individual initiatives.) Each initiative offers somewhat different and important additions to the Project's evaluation design in terms of the samples and the levels of data provided. But several features that are common across them provide unique strengths for the current reform work. First, each of these initiatives focuses on the intentional transformation of a set of schools based on a common core of reform recommendations. Second, all the initiatives and the schools in them have made the commitment to ongoing, longitudinal data collection. Evaluation efforts are viewed as integral both to the overall initiative and to the facilitation and success of local change processes. This commitment to the ongoing monitoring and assessment of implementation over a period of years overcomes a number of limitations that have plagued previous attempts to evaluate and understand school reform efforts. Prior evaluation research has typically consisted either of cross-sectional examinations of linkages between reforms and student outcomes or of assigning schools to "conditions" as a function merely of being "in" this or that initiative, without any real consideration of whether the changes sought are actually in place. For example, a recent evaluation of the impact of middle school reform in one state simply considered the achievement outcomes obtained with middle-grades students who were in schools with middle-grades configurations versus the achievement outcomes of those who were in K-8 schools or in schools with some other configuration. Not surprisingly, no differences between the two groups emerged. Further, such an analysis is a completely inadequate test of the recommendations for middle-grades reform, as there is simply no way to know the extent to which any of those schools with middle-grades configurations were engaged in even the remotest approximation of actual changes in functioning. By contrast to such evaluation models, the initiatives that have been our partners have provided us with the opportunity to monitor the degree to which schools are actually changing to reflect the recommendations of the initiative over time and to determine the association between changing levels of implementation and shifting patterns of student outcomes. A further advantage of the current Project is that in almost all these initiatives -- and certainly collectively -- the large number of schools employing common assessment measures and engaged in the transformation process has enabled us to conduct important tests of the generalizability of the process and of the impact of change across schools and student populations with quite different resources, local contexts, and backgrounds. In addition, for most of these initiatives, useful comparison data are available on schools that are not engaged in the initiatives. Indeed, in at least one instance a representative statewide sample of schools has completed the High Performance Learning Communities Assessments (described in detail below), enabling us to consider the generalizability of the data. Collectively, these aspects of the initiatives enable the Project to address a number of issues that have plagued practitioners and policy makers as they have sought to make informed decisions about reform efforts. To date those seeking data to guide such decisions have been confronted by a striking lack of large-scale empirical studies that evaluate the actual implementation of systematic education reform and its impact on student outcomes. There are a number of reasons for the gaps in the evaluation research literature on education reform. First, such work requires sustained, systematic, large-scale school reform efforts for consideration. Until recently there have been few such initiatives and even fewer that have endeavored to collect common data about the change process across their participating schools. Second, the initiatives themselves have often resisted establishing student achievement and performance as the basis of their evaluation. Indeed, in contrast to the initiatives that make up the core of this Project, others have sometimes taken the position that there is no reason to expect any correspondence between the implementation of their recommendations for reform and improvement in student achievement and performance, at least not on "traditional" standardized performance assessments. Third, the brevity of the life span of even systematic reform initiatives has presented yet another barrier to evaluation efforts. The duration of a reform initiative -- or of the funding for its evaluation -- has typically been so short that high levels of implementation and refinement fail to emerge or to be captured. At school, district, and state levels many positive practices stop before their potential is realized. These conditions and others like them have led many to conclude that the prognosis for the success of school reform is dim. Seymour Sarason notes, "It is my impression that governmental agencies and foundations are getting gun-shy about reform efforts. . . . The sense of disappointment and disillusionment is general."7 The initiatives that are the Project's partners have sought to address these gaps systematically, by giving schools extended opportunities to create change and by making exceptional commitments to the careful articulation of a testable model to guide reform, to the collection of data and rigorous evaluation, to a process for change that is built on data-based decision making (drawing on the common tool of the Project's assessments), and to accepting accountability for producing gains in student achievement. The High Performance Learning Communities Assessments The primary source of data for the Project's ongoing evaluation of its partners' reform efforts is a set of annual surveys that, collectively, makes up the High Performance Learning Communities Assessments (HiPLaCes-A). At each participating school, these surveys are administered to all teachers, staff members, students, administrators, and, at the discretion of the school, selected parents. The specific domains covered by the surveys include:
The HiPLaCes-A and their scales have undergone more than a decade of reliability and validity testing and development. During this time -- in partnership with teachers, administrators, parents, students, and policy makers and drawing on the work and scales of others -- the HiPLaCes-A have been continuously reviewed and refined to better reflect the constructs and predict the relevant student outcomes. We owe a particular debt to our practitioner partners who have spent countless hours going over wording, teaching us how to ask about key processes and practices in schools, and being constructive critics in shaping the best ways to provide school-level feedback that is maximally user-friendly and useful. The current forms of the measures have evolved through more than 1,000 school observations and through feedback from more than 10,000 teacher users and 200,000 students. They have been shown to have high levels of reliability and strong predictive and construct validity. Survey administration. All students in fourth grade and above in participating schools complete a survey. In some of the initiatives this has been a two-hour survey; in others, a one-hour short form. Teachers complete a survey that requires approximately 75 to 90 minutes. Administrators and school clerks complete surveys requiring a total of approximately two hours. Additionally, in a subset of the participating schools, teachers complete behavioral checklists on representative samples of students, and samples of parents are surveyed employing scales based on those developed by Joyce Epstein and her colleagues and others.8 Achievement/performance and academic adjustment data. Of course, the fundamental purpose of our evaluation effort is to determine the degree to which the implementation of middle-level school reform recommendations relates to the creation of schools in which all students are learning and performing at high levels. To address this concern we obtain, as available, state, local, and nationally standardized achievement, attendance, and disciplinary data on each student and school. We also obtain school descriptive data. In addition, in several of the initiatives, we have obtained much more extensive data on students, including grades, grade-level performance, special placements and honors, and so on. Qualitative and observational data. In several of the initiatives, systematic qualitative data are provided by schools and observers. In most instances these data give us converging information on either the process and levels of the implementation of specific recommendations or the overall level of progress the schools have made toward implementing Turning Points-based reforms. Further, in the MGSSPI in particular, these data have been a rich source of information about state and local policies and practices that support or impede reform, as well as about the nature of the network's functioning, which is also a critical focus of the data from Illinois. Assessment design. The HiPLaCes-A are based on a multi-measure/construct, multi-method, multi-source design. By this we mean that we are obtaining information on a variety of interrelated dimensions (or at least ones that should be conceptually interrelated), through various methods (surveys; records on grades, attendance, and disciplinary actions; reports of scores on local and state achievement tests) and with multiple sources of converging data (students, teachers, administrators, other staff members, and -- most recently -- parents). Another important feature of the assessments is their "compressed longitudinal" design. This approach is discussed in detail in the next article. The HiPLaCes School Improvement Self-Study While the Project allows for the assessment of common themes across schools and for the development of generalizable understandings of the change process, a critical goal has been to use the evaluation process to provide each participating school with school-specific feedback. The schools can then reflect on this information and use it as the basis of further monitoring, decision making, and planning. This is the Self-Study phase of the process. Once a school has completed and submitted its surveys, the Project evaluates the extent to which the changes, practices, and processes that are included in a school's improvement plan are being implemented at the school, grade, and classroom levels. Schools that have submitted data need relatively rapid feedback to determine whether they are on course and to ensure that the improvements continue to gain momentum. We have developed procedures that allow us to get back to schools within six weeks of having received their surveys with information about the effects of their improvement efforts on their students. The HiPLaCes-A Self-Study addresses several challenges that schools face in gathering data and using it effectively in planning for the future. Schools often have limited amounts of time for data gathering and careful reflection. Collecting data for the HiPLaCes-A surveys gives a structure to the process, allows schools to gather the information in an organized and comprehensive manner, and yields high-quality data that can be used for other purposes. Schools need to use the knowledge gained from the Self-Study for both short- and long-term school improvement planning. The Self-Study provides a mechanism for making important decisions by raising key questions that lead to a deeper engagement of all parties in the school's transformation. The Project does not provide information to schools with the stipulation that they must meet some specific standard. Rather, schools are asked to reflect on their current functioning or progress as it relates to locally set standards and goals. The Self-Study's results serve as a basis for continued discussion of the improvement process with a minimum of disruption to the teaching/learning process. In addition to enabling schools to more fully assess the implementation of their local school improvement plans and goals, the Project also provides procedures and training to schools so that they can use the Self-Study information to report and interpret state and local student achievement data in ways that will help them assess, over time, the overall and more specific effects of the changes they make. Results from a Longitudinal Study The longest-running and most intensive data set available to the Project has been provided by the Illinois Middle Level Network (IMLN). The staff and administrators in the founding IMLN schools and the Association of Illinois Middle Level Schools (AIMS) were instrumental in the development of the Project, the core questions, the measures, and the Self-Study procedures. Indeed, our original evaluation design was developed in response to a request from AIMS to work with that organization in the development and evaluation of the IMLN. Schools that have joined the IMLN have typically stayed in the network. This has provided us the opportunity to follow schools for a number of years as they have attempted to move through the change process; some schools in the sample are now entering their sixth year of data collection. The next article in this section describes the IMLN in detail and reports on the initial findings of the longitudinal study. It is our hope that the findings that are provided in the following article and across the life of the Project will provide policy makers and practitioners with hard evaluation research data to guide their choices from among the smorgasbord of arguments for improving education systems that confront them daily. 1. Joan Lipsitz, Successful Schools for Young Adolescents (New Brunswick, N.J.: Transaction Books, 1984). ROBERT D. FELNER is director of the National Center on Public Education and Social Policy and chair-elect of the Department of Education, University of Rhode Island, Kingston; DEBORAH KASAK is executive director of the Association of Illinois Middle Level Schools, Urbana; PETER MULHALL is a research scientist at the University of Illinois, Urbana; and NANCY FLOWERS is coordinator of research programs at the University of Illinois, Urbana. They wish to thank the state directors of the Middle Grade School State Policy Initiative, Gail Davis, and other foundation and network partners for their assistance. |
|
|
