Essential or Expendable Supports ? Assessing the Relationship between School Climate and Student Outcomes

Sociologists of education argue that school organizational practices and climates influence students’ academic outcomes. The predominant measure of school climates are aggregated student and teacher survey reports, which are diffusing into official educational statistics. Unfortunately, most studies are unable to rigorously assess the causal effects of these measures of school organization. This study does so by examining the effects of school climate experienced in grades 4–8 by different cohorts of students in Chicago Public Schools (CPS). Improvement in school climates has small positive associations with students’ eighth grade test scores and null to minimal associations with students’ chances of on-time ninth grade promotion and high school graduation.

researchers have searched for school qualities benefitting students.As Gamoran, Secada, and Marrett (2000) noted, mixed findings have frustrated these efforts, with ambiguous evidence for effects of material resources (Burtless 1996;Grubb 2009; but see Jackson, Johnson, and Persico 2016) and student composition (e.g., Gamoran 1987;Gamoran, Collares, and Barfels 2016;Lauen and Gaddis 2013).Consequently, many researchers have tried to unpack the "black box" of school effects by outlining the organizational practices leading to improved student outcomes (Gamoran et al. 2000), raising the possibility that best practices can be disseminated across schools.
Researchers use survey reports of school personnel and students to measure school organizational practices and climates.Schools where teachers report having effective principals and colleagues who take their professional roles seriously and where students report having positive relationships with teachers and learning environments free of disruption have better student outcomes and smaller inequalities in outcomes (e.g., Bodovski, Nahum-Shani, and Walsh 2013;Bryk et al. 2010;Kraft, Marinell, and Yee 2016;Moller et al. 2013).The notion of school climate is sociologically appealing.While proponents of this approach appreciate the crucial "technical core" of instruction, they draw attention to instructional quality's dependence on relationships among school actors: administrators, teachers, students, and parents (Bryk and Schneider 2002;Bryk et al. 2010).
The idea that school climate is important and quantifiable is not limited to sociologically-minded educational researchers-educational agencies are relying more and more on climate surveys.They are used in public reports on school or district quality in school districts in the United States such as Boston, Madison, New York City, and Seattle.School climate surveys are also used statewide in California, Georgia, and Illinois (among others).When the U.S. Congress renewed the Elementary and Secondary Education Act in 2015, it mandated that states incorporate nonacademic factors into their accountability systems, which in all likelihood will lead to the spread of punitive measures for schools lacking positive school climates (Blad 2016), as they are in Chicago.Moreover, a school climate industry has sprouted, with organizations such as the National School Climate Center, Panorama, and UChicago-Impact selling survey services to school systems.
Unfortunately, the collective research base cannot justify this expansion of school climate surveys.Findings on the effectiveness of school climate differ, with some studies finding either minimal or no associations (e.g., Ladd 2009;Phillips 1997) while others find very strong positive ones (e.g., Goddard et al. 2015;Moller et al., 2013).To varying degrees, these studies suffer from limited measures and limited ability to identify the causal effects of attending a school with a strong climate.

Background
Prior scholarship has emphasized eclectic, diverse, and overlapping sets of characteristics of schools' organizational climates.Syntheses by researchers at the University of Chicago Consortium on School Research (Bryk et al. 2010;Sebring et al. 2006) have been influential in drawing together disparate strands of work on school leadership, teacher professional communities, and student learning conditions.Their "essential supports" framework has seen principals as the prime movers in school improvement.Drawing on the concepts of instructional and transformational leadership (Hallinger, Bickman, and Davis 1996;Hitt and Tucker 2016;Marks and Printy 2003), Bryk and colleagues emphasized the importance of principals having a clear vision for the school, implementing high instructional standards, making sure that instruction is organized across subjects and grade levels to avoid redundancy in what students learn, providing teachers with opportunities for professional development, and fostering teacher buy-in into school improvement efforts by giving them a voice in school policies (especially over instructional and curricular matters).
For a principal to succeed in upholding high instructional standards and an organized, coherent curriculum, he or she needs to inspire teachers to embrace their roles as professionals.Sociologists have argued that the ideal school organization is communal (Lee and Smith 1993), in which teachers are willing to go above and beyond their contractual obligations and assume the responsibilities to help students learn, work with other teachers to adopt new instructional techniques, coordinate the curriculum and student assessments, and enforce rules (Bryk and Schneider 2002;Goddard et al. 2015;Moller et al. 2013).Ideally, these collective efforts, inspired by the principal, will translate into optimal pedagogy in the classroom, student engagement, and minimal disruptive behavior.
Estimated associations between school climate and student outcomes have varied across studies, and there are no clear methodological differences that explain this variation.Those with rich measures of school climate have reported no associations (Phillips 1997), modest positive ones (Johnson, Kraft, and Papay 2012;Kraft et al. 2016;Ladd 2009;Sebastian and Allensworth 2012), or very strong positive ones (Bryk et al. 2010;Goddard et al. 2015;Hallinger and Heck 2010;Ronfeldt et al. 2015).Even if attention is confined to just those showing positive effects, they suffer from shortcomings that prevent a clear conclusion that school climate matters.
First, work in this area has just started to grapple with the issue of causal inference; most studies have relied on cross-sectional associations between school climate measures and student outcomes, controlling for observed characteristics such as prior tested achievement (Goddard et al. 2015;Johnson et al. 2012;Ladd 2009;Ronfeldt et al. 2015;Sebastian and Allensworth 2012).The obvious problem with this approach is that of unobserved heterogeneity; schools with strong climates may have better student outcomes for reasons other than their climates which are not observed with control variables (including prior achievement).One likely possibility is selection bias-students predisposed to doing well academically are drawn to schools with strong climates.Teacher and student reports on school climates may be influenced by a general sense of how well the school is doing, which in turn may be colored by the academic abilities of students (Pallas 1988).
Only recently have scholars used longitudinal data on schools covering multiple cohorts of students and isolating intraschool, over-time variation in climates to study student annual gains in learning (Bryk et al., 2010:250-251;Kraft et al. 2016).They generally find small but statistically significant positive effects of measures tapping into instructional leadership and safety, with mixed results for measures of teacher collaboration and professional community.Kraft et al. (2016) argue that these effects, small as they are, are sufficiently big to the justify the use of school climate data by educators and policymakers because small effects in annual learning accumulate into big effects over time.
The second issue with school climate research is that most studies take this assumption for granted-that affecting annual (or sometimes biannual) learning gains is sufficient (Bodovski, Nahum-Shani, and Walsh 2013;Goddard et al. 2015;Hallinger and Heck 2010;Heck and Hallinger 2009;Johnson et al. 2012;Ladd 2009;Ronfeldt et al. 2015;Sebastian and Allensworth 2012).Very few studies look at other (arguably more consequential) outcomes, such as accumulated learning over a longer time period or high school graduation.This is a striking omission, given that what matters for longer-term student outcomes may not matter for test scores and vice versa (Heckman, Pinto, and Savelyev 2014;Jackson 2012;Jennings et al. 2015).The few studies looking at educational transitions-high school graduation or dropping out-tend to find modest or null effects of school climate (Bryk and Thum 1989;Kotok, Ikoma, and Bodovski 2016;Reed 2015;Rumberger and Thomas 2000).However, they are hampered by a reliance on National Center for Education Statistics (NCES) data sets, which have a limited measures of school climates, both in terms of using small samples of students and teachers to characterize an entire school and in terms of using survey questions superficially covering the aspects of climate thought to matter for school success.
In terms of accumulated learning, Moller and colleagues (2013) estimated associations with school climate and found that for black and Latino students as well as low-SES students, there are large benefits of attending schools with strong collaborative teacher communities.While this work has been useful for examining accumulated exposure to school climates, it suffers from previously mentioned shortcomings: because the authors used a NCES data set (the Early Childhood Longitudinal Study-Kindergarten cohort of 1998), they are forced to rely on sparse measures of school climate and are unable to speak to how school improvement on climate matters for student outcomes.Moreover, this study has limited school-level controls (there are no controls for school aggregates of prior tested ability and SES), raising the possibility that its strong climate effects are artifacts of unobserved heterogeneity.
This study's innovation is in combining the methodological and substantive strengths of prior work while avoiding various weaknesses.Using survey reports of most students and teachers in Chicago schools, I measure various aspects of school climate.Using longitudinal data and a specification analogous to school fixed effects models, I am able to isolate the over-time, intraschool associations between climate and student outcomes.Like Moller et al. (2013), I look at how accumulated exposure to school climates matter for student outcomes, but I go beyond accumulated learning and also examine educational transitions.

Data and Methods
The district collects annual data on all Chicago Public Schools (CPS) students' school status, demographics, and test scores and transmits them to the University of Chicago Consortium on School Research.In the early 1990s, the district, in collaboration with the Consortium (then known as the Consortium on Chicago School Research, or CCSR) fielded surveys in which students (grades 6-12) and teachers would report on various aspects of their schools' organizational, instructional, social, and academic climates. 1By the spring 1997 administration, the Consortium had introduced all survey measures used in this article and began administrating the survey on a biannual basis; in 2011, the district switched to an annual administration, although for the sake of consistency this study will only use data from the odd-numbered spring years.In the data used in this study, the student response rates ranged from 0.19 to 1.00, with a median value of 0.83 (interquartile range = 0.75-0.89),and teacher response rates ranged from 0.03 to 1.00, with a median value of 0.68 (interquartile range = 0.54-0.81).These response rates are similar to those in the initial year in the New York City data Kraft et al. (2016) used, although in New York, response rates grew over time, whereas in Chicago there is no global trend.
The population frame consists of students who were first-time third graders in the 1995-1996, 1997-1998, 1999-2000, 2001-2002, 2003-2004, and 2005-2006 school years, remained in CPS, and attended the same school in their fourth, sixth, and eighth years.For the sake of convenience, the first, third, and fifth years after being a third grader are referred to as the students' fourth, sixth, and eighth years, respectively.For outcomes related to on-time educational transitions (ninth grade promotion and high school graduation), the sample excludes students who were retained in third grade in their fourth year because it would be virtually impossible for schools to ensure timely educational transitions for such students.Because most elementary schools in CPS have a K-8 grade configuration (only a handful have a K-5 or 6-8 configuration), these measures will capture the school climates students experienced throughout their primary grade years.Table 1 shows the total number of observations and survey years for each cohort of first-time third graders.
This population frame was selected for a couple of reasons.First, because CPS students begin testing in the third grade, selecting first-time third graders allows for the controlling of student tested ability measured before they experience their school climates in their fourth, sixth, and eighth years.Second, the cohorts stop at 2005-2006 to allow for the measurement of on-time high school graduation.This choice also has the benefit of minimizing the use of school climates measured during and after the 2010-2011 school year, when school results were made public (which possibly affected teachers' answers).Third, students who attended the same school in their fourth, sixth, and eighth years were selected to isolate intraschool variation in school climates.In an alternative specification presented in the supplement, this requirement will be relaxed, at the cost of allowing for interschool variation in school climates.

Outcomes
I analyze four student outcomes.Graduation is an indicator if the student graduated from a CPS high school nine years years after being a first-time third grader; ontime ninth grade promotion is an indicator if the student was in the ninth grade six years after being a first-time third grader; math and reading eighth-grade scores are scores from standardized tests given to students in the spring of their eighth grade year, which could be the Iowa Test of Basic Skills (ITBS), the original version of the Illinois Standards Achievement Test (ISAT), and a revised version of the ISAT produced in response to No Child Left Behind (NCLB).Scores on these tests were standardized by grade level and test; students who took multiple tests had their scores averaged. 2Summary statistics on these outcomes, as well as predictors used in this analysis, are presented in Table 2.Note that z-standardization for test scores and other variables was done for the entire CPS population, but the mean values presented in Table 2 are substantially different from zero, reflecting the fact that analyses are restricted to students who remained in the same school in their fourth, sixth, and eighth years.

School Climate
For the 1997-2013 period, the Consortium has comparable estimates of 16 different aspects of school organizational climate; 11 are from the teacher surveys and five are from the student surveys.Each of these measures are listed in Table 3, along with a representative question (each measure is based on at least four different survey questions).I constructed three different "omnibus" measures: an overall climate measure (Cronbach's alpha = 0.91) that is the average of all measures, an average teacher climate measure (Cronbach's alpha = 0.91), and an average student climate measure (Cronbach's alpha = 0.72). 3 The Consortium was thorough and careful in developing and refining these measures, informed by the essential supports framework.A Consortium psychometrician used Rasch analysis (Bond and Fox 2015) to test the appropriateness of these measures from earlier survey administrations (in 1991 and 1994) and to guide their further refinement.Rasch analysis estimates a difficulty level for each item (the fewer respondents endorsing an item, the more "difficult" it is) and an ability level for each respondent (the more respondents endorse items, particularly difficult ones, the higher his or her "ability" is).Measures based on items that can separate respondents based on ability have higher reliability.This approach is more sophisticated than common data reduction techniques used by sociologists (i.e., summed scales and factor analysis) because it forces analysts to grapple with the problem of being able to distinguish a wide range of respondent abilities.An additional advantage of Rasch measures is that measurement can change over time (items can be added or removed), but the estimates are comparable over time.
Within each year, school climate measures are constructed by aggregating all individual-level measures at the school level (each individual-level measure is weighted by the inverse of its standard error, such that individuals with more reliable measures are weighted more). 4For each student, a school climate measure was constructed by averaging their climate observations in their fourth, sixth, and  Most students in my school like to put others down eighth years in CPS.For a student whose school participated in the survey all three years, his or her measure would be constructed thusly: where j indexes the school the student attended in his or her fourth, sixth, and eighth years, c indexes his or her cohort (the year he or she was a first-time third grader), and t indexes the year a school's measure comes from.For example, measure jt=c+3 is a school climate measure, aggregated from all individual responses in school j three years after students in cohort c were third graders (their sixth-year school).Students whose school participated in the survey only one time or zero times were dropped from the analysis. 5All of these measures are z-standardized at the school-cohort level.

Student Controls
To account for potential student selection into schools, I control for various student characteristics.Student FRPL status is the student's average free/reduced-price lunch status, coded 1 if the student received free lunches, 0.5 if the student received reduced-price lunches, and zero if the student received neither.Student neighborhood disadvantage is an index of the poverty rate and joblessness rate in the block group in which the student lived (these proportions were converted to logits and z-standardized).Student neighborhood advantage is the mean years of education of individuals living in the student's block group.All three of these socioeconomic status variables-FRPL status, neighborhood disadvantage, and neighborhood advantage-are averaged across all years in which the student was in the CPS system.Student race consists of dummy indicators for being black; Latino; white/Asian; and other/missing race, while student sex is a dummy indicator for being male.Bilingual education status is measured by whether or not the student ever received bilingual education services when he/she was in CPS.Age is the student's age in years on January 1st in the year in which the student was a first-time third grader.Linear and quadratic terms for age are entered into the models.Student third grade math and reading test scores are constructed the same as eighth grade scores (see footnote 2). 6

School Controls
To ensure that school climate is not confounded with other characteristics of schools, various time-varying demographic characteristics of schools are controlled for.School averages were constructed for FRPL status, neighborhood socioeconomic disadvantage, neighborhood socioeconomic advantage, and third grade math and reading scores.Other school-level controls include racial composition (proportion black; proportion Asian/white, and proportion other/missing), the natural log of enrollment size, and charter status.All school-level demographic variables are treated the same way as the school climate measures-the average of the student's fourth-year, sixth-year, and eighth-year observations is calculated.

Analytic Strategy
This study pools data on six cohorts of first-time third graders in CPS.Logistic regression was used to analyze the high school graduation and on-time ninth grade promotion outcomes; linear regression was used to analyze eighth grade test scores.
For both kinds of models, a school random effect was added to reduce bias in estimates of standard errors and to increase the efficiency of parameter estimates (Raudenbush and Bryk 2002).
As previously mentioned, unobserved heterogeneity afflicts studies using observational data on school climates.As with most other studies, I cannot completely eliminate the problem of unobserved heterogeneity, but I attempt to minimize it by following the steps of Kraft et al. (2016) and running a model akin to adding a fixed effect for schools.By controlling for the school a student attends and leveraging only over-time, intraschool variation in school climates, I account for all time-invariant school characteristics and presumably most family-and student-level characteristics that lead students to select into particular schools.I will return to this point when I partition variation in school climates.Of course, even if selection bias could be completely removed, a fixed effects specification would not guarantee causal identification because a school fixed effects model would not account for time-varying, school-level unobserved variables.
Unfortunately, because sample sizes range from 50,000-70,000 cases, it was computationally impossible to run a fixed effects specification with logistic regressions.I get around this issue by using what is alternatively called the "hybrid" (Allison 2009) or "between-within" (Sjölander et al. 2013) method.This approach simulates a fixed effects model by running a random effects regression and centering all predictors on the school mean (as well as controlling for the school means themselves): where outcome Y for student i in school j and cohort c is a function of her school's mean climate measure (measure j ), her school-cohort's deviation from the school mean (measure jc − measure j ), vectors of student-level variables X and school-cohortlevel variables W (that are similarly partitioned into school means and deviations from the school mean), school random effect U j , and a student-level error term e ijc .The coefficient for the centered versions of school climate measures represent the fixed effects estimates and are the only result presented from the between-within model. 7

Variation among Schools and Variation among Cohorts within Schools
A potential concern with this study's design is that differences (in either outcomes or school climates) among cohorts attending the same school is a poor source of variation.To address this concern, multilevel linear models were run to partition the variances of outcomes and school climates. 8For outcome measures, three-level models were run, with students nested in their school-cohorts that in turn were nested in their schools; for the climate measures, two-level models were run, with school-cohorts nested in schools.The results are presented in Table 4. "Unadjusted" standard deviations and intraclass correlations (ICCs) are presented from empty models with no predictors; "adjusted" statistics are also presented from models including all student-level controls (for the climate measures, these controls are aggregated at the school-cohort level).The main conclusion from Table 4 is that there is substantial variation among cohorts attending the same school, both in terms of outcomes and in terms of school climate.After adjusting for student characteristics, the standard deviation of cohorts from their school mean in terms of test scores is about 0.12 (for reading) and 0.19 (for math) SD units; for on-time ninth grade promotion and high school graduation, the standard deviation is about 5 percent; for school climates, the standard deviation hovers between 0.62 and 0.71 SD units.
However, this selection is not particularly responsive to over-time changes in school climates, as can be seen by the fact that intraschool, intercohort variances in school climates are reduced by only around 15 percent (e.g., [0.67 2 -0.62 2 ] / 0.67 2 = 0.14) after controlling for student characteristics.And in turn, student characteristics explain substantially less of the intercohort variances in student outcomes: 23 percent of eighth grade test scores (e.g., [0.22 2 -0.19 2 ] / 0.22 2 = 0.23) and 30 percent of on-time ninth grade promotion ([6.0% 2 -5.0% 2 ] / 6.0% 2 = 0.30).High school graduation is a clear exception: student characteristics explain 71 percent ([8.4% 2 -4.5% 2 ] / 8.4% 2 = 0.71) of intercohort variance in this outcome.If we make the (admittedly untestable) assumption that unobserved characteristics affecting selection have similar distributions and effects as observed characteristics, then selection bias should be much less of a problem when examining the associations between within-school changes in school climate, on the one hand, and test scores and on-time ninth grade promotion on the other.

Missing Values and Attrition
The population frame of this study is all first-time third graders in CPS in 1995-1996, 1997-1998, 1999-2000, 2001-2002, 2003-2004, and 2005-2006.Table S.5 in the online supplement lists the total number of students potentially eligible to be in the study, the number of students dropped for various reasons, and the number of students who ended up in the sample.
Between 33 and 39 percent of first-time third graders are retained for analyses.The biggest cause of missing cases are students who stayed in CPS and have complete data but transferred to a different CPS school in their sixth or eighth year.Such students account for between 27-32 percent of first-time third graders.In an alternative specification (discussed in the online supplement), they are retained, and the results are similar to the main set presented in this paper.The second biggest cause of missing data are students who have missing outcomes, usually resulting from transferring out of CPS by the time the outcome was measured.Such cases account for 19-31 percent of first-time third grade students.Analyses of attrition (see Table S.6 in the online supplement) show that school climate itself has small and contradictory effects on attrition.Instead, the main causes of attrition are race and socioeconomic status.Asian/white students and students who come from non-poor families or advantaged neighborhoods are much more likely to transfer out of CPS.As will be seen, the associations between school climate and student outcomes are somewhat stronger for these groups, so it is possible there is some downward bias in the presented estimates of school climate.This bias should be minimal because these students constitute only a small share of CPS first-time third graders (5 percent spent zero years receiving free or reduced-price lunches; 12 percent are Asian or white).
The other sources of missing data account for only a small portion of first-time third graders.Less than 10 percent of students are deleted because their school participated in the surveys either not at all or only once, and 1 percent of students are deleted because of missing data on control variables.

Associations between School Climate and Student Outcomes
Table 5 presents the associations between the omnibus climate measures and student outcomes.For each outcome, I present the results from a regular school random effects model (in which no predictors are recentered) and a random effects "betweenwithin" model in which all predictors are centered on their school means.For test scores, I present fully standardized coefficients; for on-time ninth grade promotion and high school graduation, I present average marginal effects (AMEs).Supplement Table S.7 presents the associations between individual school climate measures and student outcomes.Because the data are populations and not random samples of CPS third-grade cohorts, I will not refer to the statistical significance of coefficients, although for the sake of comparison with other studies, tables include information on coefficients' standard errors and significance levels.School climate measures have positive but modest associations with eighth grade test scores in both the regular and between-within random effects models.In fact, the coefficients for climate tend to be larger in the between-within model than in the regular random effects model.In the between-within models, a 1 SD increase in overall school climate is associated with students' eighth grade math and reading scores by 0.05 and 0.04 SD, respectively.If these associations are standardized using the standard deviations for within-school, intercohort variation presented in Table 4, more impressive standardized coefficients of 0.15 (0.05 * 0.67 / 0.22) for math and 0.19 (0.04 * 0.67 / 0.14) for reading are obtained.
The results for on-time ninth grade promotion are more disappointing.Overall climate, student climate, and teacher climate have virtually null associations with on-time ninth grade promotion.
The school climate measures also have minimal associations with high school graduation; all of the omnibus measures of climate have null associations in the regular random effects model.In the between-within model, the climate measures grow in magnitude, with AMEs of 0.01.As with test scores, these AMEs could be standardized to represent within-school, intercohort variation, producing effect sizes of 0.08 (0.01 * 0.67 / 0.084) for overall climate, 0.09 (0.01 * 0.74 / 0.084) for student climate, and 0.08 (0.01 * 0.70 / 0.084) for teacher climate.
In the supplement, I relax the assumption that the associations between school climate and student outcomes are linear.The analyses show some minor nonlinear associations, particularly those between teacher climate measures and student test scores (improvement in teacher climate yields somewhat of a larger benefit for schools that tend to have weak teacher climate measures).I also test for the possibility that school climates experienced earlier in students' educational careers may matter more than those experienced in their later years and find no evidence for it.In addition, in the supplement I test for various threats to the conclusion that school climates have minimal-to-null associations with student outcomes.I find that this conclusion holds up even under alternative specifications addressing these threats.

Are the Associations Moderated by Student Characteristics?
It is possible that school organizational climate is stronger for some students than others.Lee, Smith, and Croninger (1997) argue that students from disadvantaged backgrounds are especially worse off when they attend schools characterized by impersonal student-teacher relations and low teacher expectations of students.Moller et al. (2013) also provide evidence that teacher professional communities are more beneficial for racial minority students and students of low socioeconomic status (SES).It is possible that the small or nonexistent associations between school climate and student outcomes shown thus far are an artifact of heterogeneity in climate's effects.
To test this possibility, I reran the between-within models, and in separate models I included interaction terms between the overall school climate measures with student sex, FRPL status, race, and third grade test scores.I present conditional associations in Table 6.There is not much evidence for Lee and Smith's (1997) and Moller et al.'s (2013) contentions that school climates especially matter for vulnerable students.To the extent there are differences in the associations, they tend to occur in favor of more advantaged students-the associations with overall climate are more positive for students who never received free or reduced price lunches (for test scores and graduation).For advocates of measuring school climates and organizational practices using surveys, there is good news and there is bad news.The good news is that there are associations between school climate and accumulated student learning, and these associations are probably not an artifact of student selection into schools.Of course, it is impossible to rule out a scenario in which they are an artifact of an unmeasured, time-varying school-level confound, but given the rich school-level demographic controls used in this study, it seems unlikely.
The bad news is, first, these associations are fairly small.Second, they appear to be stronger for students who come from families with more resources (namely students who do not received free or reduced price lunches).The idea that improving school climates can help schools overcome the problems with disadvantaged student populations is not supported.Third, there is no association with on-time ninth grade promotion.Fourth, the fact that school climate has null to minimal associations with graduation and ninth grade promotion also casts doubt on the importance of the associations with test scores.What good is it to score 0.05 standard deviations higher on an eighth-grade test when a student is no more likely to be promoted to the ninth grade on time or graduate from high school?
The findings of this study have precedent in previous research showing small associations between school climate measures and test scores (Kraft et al. 2016;Ladd 2009;Sebastian and Allensworth 2012) and minimal to null associations with high school graduation outcomes (Bryk and Thum 1989;Reed 2015;Rumberger and Thomas 2000).Of these studies, Kraft et al.'s (2016) is the only one to leverage within-school variation in school climates; like this one, they found that isolating the within-school, over-time effects of school climate mattered little for the estimated effects.
Other studies have found more substantial effects on test scores, with effect sizes exceeding 0.20 (Goddard et al. 2015;Hallinger and Heck 2010;Ronfeldt et al. 2015).This study notably contradicts the findings of two prior studies using data on CPS schools.A brief appendix in Bryk et al. (2010:250-251) used the same climate measures as this study and a similar time range in Chicago (1997Chicago ( -2005)), and found large between-and within-school effects of school climate measures (exceeding standardized coefficients of 0.30 and 0.10, respectively) on school's value-add to test scores.While these studies used careful designs, either controlling for prior achievement or looking at students' achievement trajectories, I speculate their large effect sizes are due to a combination of (a) having the school as the unit of analysis and thus inflating effect sizes by removing within-school, between-student variation from the outcome and (b) measurement error in their control variables for student SES.I went to some length to minimize measurement error in SES by controlling for both FRPL status and students' block group characteristics as well as making both variables averages across all years each student was in CPS.
Additonally, a report by this author (Klugman et al. 2015) found that school climate measures had strong associations with graduation rates in CPS high schools (but not in other Illinois public high schools).That report used only school-level, cross-sectional data and could not include the student-level covariates related to prior achievement that I controlled for in this study.
Some sociologists argue that attacking inequalities among families is the only viable route to ameliorating educational inequalities (e.g., Downey and Condron 2016).Like the results of Jennings et al. (2015), my findings indicate that this argument is too quick to give up on school improvement, given that schools' student outcomes do change substantially over time in ways that are not reducible to changing demographics (Table 4).However, this study's main conclusion is that assessing schools' organizational practices and climates using surveys is not a feasible means to achieve school improvement.
There are two possible interpretations of this finding.One, the sociological insight that relationships among school actors influence the technical core of schoolsinstruction-and thus student outcomes is correct, but surveys, for all of their practicality and economy, cannot capture the quality of these relationships with much precision.
Two, the measures are sufficiently precise, but the sociological contention about school relationships is overstated.Schools are "loosely coupled" systems (Weick 1976), so the quality of teachers' work is dependent just on their individual efforts to convey knowledge to students and inspire them to meet high academic standards (Lortie 1977).Broader sets of relationships, beyond that of the student-teacher in the classroom, are just not consequential for school productivity.
This study cannot adjudicate between these two scenarios, but there are reasons for according more weight to imprecise survey measures than to the loose coupling scenario.There is evidence that interventions focusing on improving relationships among teachers and administrators is consequential for school improvement (Borman et al. 2003;Cook, Murphy, and Hunt 2000).Notably, Cook et al. (2000) found that in Chicago, the Comer School Development Program, which targets relationships among teachers and administrators, was efficacious in improving student outcomes and survey reports of certain aspects of school climate, but improved school climate did not explain the improvement in student outcomes-in other words, the Comer schools seeing improved outcomes did not see improved survey reports of school climate, and vice versa.
The loose coupling scenario focusing on the individual efforts of teachers is arguing that aggregated survey reports capturing different concepts, those related to teachers' efforts in the classroom, should affect student outcomes.For instance, the Consortium has deployed new survey questions administered to students, asking them about their schools' academic press (Murphy et al. 1982) as well as the pedagogical techniques used in their classes.Unfortunately, they were not deployed consistently through this study's time frame and could not be used in these analyses.Surely they would matter more for student outcomes?This is doubtful.It is striking that for all of the Consortium's attempts to capture distinct aspects of school organization, their different measures end up having similar associations with outcomes (as shown in Table S.7 in the online supplement).This is consistent with a scenario that survey reports are based on teachers' and students' global sense of how well the school is doing.In that case, it is unlikely there is some missing concept that needs to be uncovered to show strong climate effects.
It is customary for social science articles to conclude with calls for more research.And indeed, confirmation or disconfirmation of this study's findings should be pursued (and can be easily achieved with existing data).But if the findings are confirmed, the major efforts to collect and analyze school climate data undertaken by districts, states, various not-for-profit and for-profit entities, as well as the time taken by students and teachers to complete the questionnaires, have been wasteful and should cease.

Notes
7 Unfortunately, the between-within method introduces a modest amount of bias in logistic regressions (Allison 2014;Brumback et al. 2010;Goetgeluk and Vansteelandt 2008).Per Allison's (2014) advice, I checked for this bias by comparing the between-within results to another between-within model with added controls for the squared and cubic terms of the school means for all predictors, as well as the school standard deviation of all predictors.The results are nearly identical, indicating bias is not a concern.
8 Multilevel linear probability models were run for the dichotomous outcomes of ontime ninth grade promotion and high school graduation for the sake of having easily interpretable standard deviations and intraclass correlations.

Table 1 :
Third-grade cohorts used in analyses.

Table 2 :
Summary statistics of all variables used in analyses.

Table 3 :
Description of school climate measures.

Table 4 :
Variance partitioning of outcomes and school climate measures.
Notes: Outcome measures were analyzed in three-level multilevel linear models, with students nested in school-cohorts nested in schools.Climate measures were analyzed in two-level multilevel linear models, with school-cohorts nested in schools.The adjusted model controls for student sex, race, cohort, age at third grade, bilingual education status, FRPL status, neighborhood advantage, neighborhood disadvantage, and third grade test scores.For school climate measures, these were aggregated at the school-cohort level.

Table 6 :
Associations between overall school climate and student outcomes, moderated by student characteristics.Standard errors are in parentheses and 95 percent confidence intervals are in brackets.All coefficients are for the school-centered measures from separate between-within models.Models include all student-and school-level controls.Survey measures of schools' organizational practices and climates are common in educational research and they are diffusing into official educational statistics in various states and localities.Unfortunately, prior research has only just begun to go beyond controlling for observables when analyzing the associations of school climate with student outcomes.This study uses rich, longitudinal measures of school climate and rigorous methods to estimate associations between school climate and accumulated learning from third grade to eighth grade, students' on-time promotion to ninth grade, and high school graduation.