Search EdWorkingPapers by author, title, or keywords.
Researchers use test outcomes to evaluate the effectiveness of education interventions across numerous randomized controlled trials (RCTs). Aggregate test data—for example, simple measures like the sum of correct responses—are compared across treatment and control groups to determine whether an intervention has had a positive impact on student achievement. We show that item-level data and psychometric analyses can provide information about treatment heterogeneity and improve design of future experiments. We apply techniques typically used in the study of Differential Item Functioning (DIF) to examine variation in the degree to which items show treatment effects. That is, are observed treatment effects due to generalized gains on the aggregate achievement measures or are they due to targeted gains on specific items? Based on our analysis of 7,244,566 item responses (265,732 students responding to 2,119 items) taken from 15 RCTs in low-and-middle-income countries, we find clear evidence for variation in gains across items. DIF analyses identify items that are highly sensitive to the interventions—in one extreme case, a single item drives nearly 40% of the observed treatment effect—as well as items that are insensitive. We also show that the variation of item-level sensitivity can have implications for the precision of effect estimates. Of the RCTs that have significant effect estimates, 41% have patterns of item-level sensitivity to treatment that allow for the possibility of a null effect when this source of uncertainty is considered. Our findings demonstrate how researchers can gain more insight regarding the effects of interventions via additional analysis of item-level test data.
This paper conceptualizes segregation as a phenomenon that emerges from the intersection of public policy and individual decision-making. Contemporary scholarship on complex decision-making describes a two-step process—1) Editing and 2) Selection— and has emphasized the individual decision-maker’s agency in both steps. We build on this work by exploring, both theoretically and empirically, how policy can structure the choices individuals face at each step. We conduct this exploration within the empirical context of enrollment decisions among families in the Wake County Public School System (WCPSS), which used a controlled school choice system to help achieve diversity aims. We first investigate the schooling choice sets that WCPSS constructed for families and then examine families’ schooling selections. We find that families were offered choice sets containing schools varying racial compositions, but that the racial makeup of schools in families’ choice set systematically varied by schooling type and student race/ethnicity. We further show that a majority of families enrolled in their district-assigned default school, with Black and Hispanic families more likely than white or Asian families to attend this option. Finally, we demonstrate that white or Asian families enroll in their default school at lower rates as the share of Black students increases.
Challenging the conventional wisdom that the spread of democracy was a leading driver of the expansion of primary schooling, recent studies show that democratization in fact did not lead to an average increase in primary school enrollment rates. One reason for this null effect is that there was already considerable provision of primary education before democratization. Still, it is possible that the spread of democracy did impact other aspects of education systems, such as the content of education and the extent to which teaching jobs are politicized. Studying this possibility cross-nationally has been infeasible due to data limitations. To address this gap, we take advantage of an original dataset covering 160 countries from 1945 to 2021 that contains information about these aspects of education. We document that transitions to democracy tend to be preceded by a decline in the politicization of both education content and teaching jobs. However, soon after democratization occurs, this decline usually halts. Counterfactual estimates suggest that democratization roughly halves the degree to which teacher hiring and firing decisions are politicized, but has a smaller impact on the content of education. The empirical patterns that we uncover have important implications for future research.
How progressive is school spending when spending is measured at the school-level, instead of the district-level? We use the first dataset on school-level spending across schools throughout the United States to ask to what extent progressivity patterns previously examined across districts are amplified, nullified, or reversed, upon disaggregation to schools. We find that progressivity is systematically greater when we conduct a school-level analysis, rather than district-level analysis. This may be surprising, given the traditional view in public economics that local governments cannot effectively redistribute. We thus probe the data for explanations for this pattern, uncovering evidence that federal policies play an important role in driving within-district progressive allocations. In particular, we can explain about 83% of the within-district contribution to progressivity by the federal component of spending plus allocations that are empirically attributable to special education and English language learning programs. Our findings are thus consistent with the traditional view of redistribution being primarily the purview of central governments, operationalized in this context through mandates.
This is one of the first studies of the mismatch between students’ test scores and teachers’ estimations of those scores in low- and middle-income countries. Prior studies in high-income countries have found strong correlations between these metrics. We leverage data on actual and estimated scores in math and language from India and Bangladesh and find that teachers misestimate their students’ scores and that their estimations reveal their misconceptions about students in most need of support and variability within their class. This pattern is partly explained by teachers’ propensity to overestimate the scores of low-achieving students and to overweight the importance of intelligence. Teachers seem unaware of their errors, expressing confidence in estimations and surprise about their students’ performance once revealed.
A controversial, equity-focused mathematics reform in the San Francisco Unified School District (SFUSD) featured delaying Algebra I until ninth grade for all students. This descriptive study examines student-level longitudinal data on mathematics course-taking across successive cohorts of SFUSD students who spanned the reform’s implementation. We observe large changes in ninth and tenth grades (e.g., delaying Algebra I and Geometry). Participation in Advanced Placement (AP) math initially fell 15% (6 pp.) driven by declines in AP Calculus and among Asian/Pacific-Islander students. However, growing participation in acceleration options attenuated these reductions. Large ethnoracial gaps in advanced math course-taking remained.
The effects of the COVID-19 pandemic on students’ experiences in school were widespread. Early research show reductions in test scores across grade levels and student groups. This study extends research evidence to additional student outcomes – absences, course grades, and grad retention – and to examine how pandemic effects are distributed across students. Using a combination of descriptive and regression analyses, we find negative average impacts on all outcomes. These effects are largest at the high end of the absence distribution and the low end of the grade distribution. Effects are also largest in middle school for most outcomes and are typically larger among historically marginalized groups of students. These findings reflect widening achievement gaps and the need for targeted supports.
Prior research has documented substantial inequity across, racial, ethnic, and socioeconomic lines within the population of students identified as gifted. Less attention has paid to the equity of gifted identification for student learning English or those with disabilities and what effect state policies toward gifted education might have on these rates. This paper attempted to fill that void by analyzing data from the Office of Civil Rights Data Collection and Stanford Education Data Archive along with original coding of state gifted education policies. Our findings show that while both groups are substantially underrepresented, state mandates for schools to offer services, requirements for formal gifted education plans, and regular audits for compliance are correlated with much higher rates of gifted service availability and equity for English learners and students with disabilities. We also describe the location and characteristics of the top 5% most equitable schools for English learners and students with disabilities.
We provide evidence from a randomized controlled trial on the effectiveness of a novel, 100-percent online math tutoring program, targeted at secondary school students from highly disadvantaged neighborhoods. The intensive, eight-week-long program was delivered by qualified math teachers in groups of two students during after-school hours. The intervention significantly increased standardized test scores (+0.26 SD) and end-of-year math grades (+0.48 SD), while reducing the probability of repeating the school year. The intervention also raised aspirations, as well as self-reported effort at school.
School autonomy has been and continues to be one of the most important education reform strategies around the world despite ambiguity about its theoretical and empirical effects on students learning. We use international data from PISA to test three country-level factors that might account for inconsistent results in prior literature: (1) the selective implementation of school autonomy based on school performance; (2) differential influence on high-risk subgroups; and (3) the presence of accountability policies to prevent opportunism by autonomous schools. We find that the relationship between autonomy and student test performance varies both across countries and within countries across subgroups in both magnitude and direction. Similar results are observed if decentralization is coupled with accountability policies. All of three tested factors influence country-level associations between school decentralization and student learning, which suggests that autonomy is effective only when contextual factors and other policies are aligned.