Search EdWorkingPapers

Search EdWorkingPapers by author, title, or keywords.

EdWorkingPapers

Angela Johnson, Megan Kuhfeld, James Soland.

Nearly one in five U.S. students attends a rural school, yet we know very little about achievement gaps and academic growth in rural schools. This study leverages a unique dataset that includes longitudinal test scores for more than five million 3rd to 8th grade students in approximately 17,000 public schools across the 50 states, including 900,000 students attending 4,727 rural schools. We find rural achievement and growth to be slightly above public schools. But there is considerable heterogeneity by student race/ethnicity. For all grades and subjects, White-Black and White-Hispanic gaps are smaller in rural schools than gaps nationwide, and White-Native American gaps are larger in rural schools than gaps nationwide. Separate analyses by racial/ethnic subgroup show that rural Black, Hispanic, and Native American students are often growing slower than their respective subgroup national average. In contrast, White students are often growing faster than the national average for White students.

More →


Lindsay C. Page, Matthew A. Lenard, Luke Keele.

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practice for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in Wake County, North Carolina.

More →


Luke Keele, Matthew A. Lenard, Lindsay C. Page.

Many interventions in education occur in settings where treatments are applied to groups. For example, a reading intervention may be implemented for all students in some schools and withheld from students in other schools. When such treatments are non-randomly allocated, outcomes across the treated and control groups may differ due to the treatment or due to baseline differences between groups. When this is the case, researchers can use statistical adjustment to make treated and control groups similar in terms of observed characteristics. Recent work in statistics has developed matching methods designed for contexts where treatments are clustered. This form of matching, known as multilevel matching, may be well suited to many education applications where treatments are assigned to schools. In this article, we provide an extensive evaluation of multilevel matching and compare it to multilevel regression modeling. We evaluate multilevel matching methods in two ways. First, we use these matching methods to recover treatment effect estimates from three clustered randomized trials using a within-study comparison design. Second, we conduct a simulation study. We find evidence that generally favors an analytic approach to statistical adjustment that combines multilevel matching with regression adjustment. We conclude with an empirical application.

More →


Allison Atteberry, Andrew McEachin.

Summer learning loss (SLL) is a familiar and much-studied phenomenon, yet new concerns that measurement artifacts distorted canonical SLL findings create a need to revisit basic research on SLL. Though race/ethnicity and SES only account for about 4% of the variance in SLL, nearly all prior work focuses on these factors. We zoom out to the full spread of differential SLL and its contribution to students’ positions in the eighth grade achievement distribution. Using a large, longitudinal Northwest Evaluation Association dataset, we document dramatic variability in SLL. While some students actually maintain their school-year learning rate, others lose nearly all their school-year progress. Moreover, decrements are not randomly distributed—52% of students lose ground in all 5 consecutive years (ELA).

More →


Jonathan Smith, Joshua Goodman, Michael Hurwitz.

We provide the first estimated economic impacts of students’ access to an entire sector of public higher education in the U.S. Approximately half of Georgia high school graduates who enroll in college do so in the state’s public four-year sector, which requires minimum SAT scores for admission. Regression discontinuity estimates show enrollment in public four-year institutions boosts students’ household income around age 30 by 20 percent, and has even larger impacts for those from low income high schools. Access to this sector has little clear impact on student loan balances or other measures of financial health. For the marginal student, enrollment in such institutions has large private returns even in the short run and positive returns to state budgets in the long run.

More →


Angela Johnson, Diana Mercado-Garcia.

English Learners (ELs) lag behind their peers in postsecondary attainment. As the EL population in the U.S. continues to grow, so does concern over their underrepresentation in higher education. Research shows that Early College High Schools have a significant impact on high school and college outcomes for students from low income and racial/ethnic minority backgrounds, but how similar opportunities might extend to ELs remains unknown. We report findings from the first three years of an intervention that offers Early College opportunities in high schools serving large EL populations. Leveraging an exogenous policy change and rich administrative records, we examine the outcomes of pre- and post-program cohorts of ELs (N=15,090) in treated and untreated high schools. We find a large, significant impact on the number of college credits earned in 12th grade but no effect on immediate college attendance after high school. The probability of attending a four-year college significantly decreased.

More →


Angela Johnson, Megan Kuhfeld, Greg King.

This study presents a framework that uses academic trajectories in the middle grades for identifying students in need of intervention and providing targeted support. We apply a set of academic college readiness benchmarks to rich longitudinal data for more than 360,000 students in 5900 schools across 49 states and the District of Columbia. In both math and reading, each student was assessed up to six times (fall and spring of 6th, 7th, and 8th grade). We show that student-level and school-level demographic characteristics significantly predict academic trajectories. Compared to White and Asian students, higher proportions of Black and Hispanic student are consistently off-track for college readiness throughout middle school. Among students who started 6th grade on track, being male, Black, Hispanic, and attending schools with a higher percentage of students who are eligible for free or reduced-price lunch are positively associated with falling off track.

More →


Jihyun Kim, Ken Frank, Peter Youngs, Serena Salloum, Kristen Bieda.

While teacher evaluation policies have been central to efforts to enhance teaching quality over the past decade, little is known about how teachers change their instructional practices in response to such policies. To address this question, this paper drew on classroom observation and survey data to examine how early career teachers’ (ECTs’) perceptions of pressure associated with teacher evaluation policies seemed to affect their enactment of ambitious mathematics instruction. As part of our analysis, we also considered the role that mathematical knowledge for teaching (MKT) and school norms regarding teaching mathematics shape the potential influence of teacher evaluation policies on ECTs’ instructional practices. Understanding how the confluence of these factors is associated with teachers’ instruction provides important insights into how to improve teaching quality, which is one of the most important inputs for student learning.

More →


Adam Altmejd, Andres Barrios-Fernandez, Marin Drlje, Joshua Goodman, Michael Hurwitz, Dejan Kovac, Christine Mulhern, Christopher Neilson, Jonathan Smith.

Family and social networks are widely believed to influence important life decisions but identifying their causal effects is notoriously difficult. Using admissions thresholds that directly affect older but not younger siblings’ college options, we present evidence from the United States, Chile, Sweden and Croatia that older siblings’ college and major choices can significantly influence their younger siblings’ college and major choices. On the extensive margin, an older sibling’s enrollment in a better college increases a younger sibling’s probability of enrolling in college at all, especially for families with low predicted probabilities of enrollment. On the intensive margin, an older sibling’s choice of college or major increases the probability that a younger sibling applies to and enrolls in that same college or major. Spillovers in major choice are stronger when older siblings enroll and succeed in more selective and higher-earning majors. The observed spillovers are not well-explained by price, income, proximity or legacy effects, but are most consistent with older siblings transmitting otherwise unavailable information about the college experience and its potential returns. The importance of such personally salient information may partly explain persistent differences in college-going rates by geography, income, and other determinants of social networks.

More →


Megan Kuhfeld, James Soland, Beth Tarasawa, Angela Johnson, Erik Ruzek, Jing Liu.

With 55 million students in the United States out of school due to the COVID-19 pandemic, education systems are scrambling to meet the needs of schools and families, including planning how best to approach instruction in the fall given students may be farther behind than in a typical year. Yet, education leaders have little data on how much learning has been impacted by school closures. While the COVID-19 learning interruptions are unprecedented in modern times, existing research on the impacts of missing school (due to absenteeism, regular summer breaks, and school closures) on learning can nonetheless inform projections of potential learning loss due to the pandemic. In this study, we produce a series of projections of COVID-19-related learning loss and its potential effect on test scores in the 2020-21 school year based on (a) estimates from prior literature and (b) analyses of typical summer learning patterns of five million students. Under these projections, students are likely to return in fall 2020 with approximately 63-68% of the learning gains in reading relative to a typical school year and with 37-50% of the learning gains in math. However, we estimate that losing ground during the COVID-19 school closures would not be universal, with the top third of students potentially making gains in reading. Thus, in preparing for fall 2020, educators will likely need to consider ways to support students who are academically behind and further differentiate instruction.

 

More →