Program and policy effects

While policymakers have demonstrated considerable enthusiasm for “science of reading” initiatives, the evidence on the impact of related reforms when implemented at scale is limited. In this pre-registered, quasi-experimental study, we examine California’s recent initiative to improve early literacy across the state’s lowest-performing elementary schools. The Early Literacy Support Block Grant (ELSBG) provided teacher professional development grounded in the science of reading as well as aligned supports (e.g., assessments and interventions), new funding (about $1000 per student), spending flexibility within specified guidelines, and expert facilitation and oversight of school-based planning. We find that ELSBG generated significant (and cost-effective) improvements in ELA achievement in its first two years of implementation (0.14 SD) as well as smaller, spillover improvements in math achievement.

More →

High-quality preschool programs are heralded as effective policy solutions to promote low-income children’s development and life-long wellbeing. Yet evaluations of recent preschool programs produce puzzling findings, including negative impacts, and divergent, weaker results than demonstration programs implemented in the 1960s and 70s. We provide potential explanations for why modern preschool programs have become less effective, focusing on changes in instructional practices and counterfactual conditions. We also address popular theories that likely do not explain weakening program effectiveness, such as lower preschool quality and low-quality subsequent environments. The field must take seriously the smaller positive, null, and negative impacts from modern programs and strive to understand why effects differ and how to improve program effectiveness through rigorous, longitudinal research.

More →

Recent work highlights the challenge of scaling evidence-based strategies to achieve social policy objectives. We evaluate, through a randomized control trial, a national financial incentive program designed to increase student engagement with college advising and completion of college and financial aid milestones that prior experimental studies demonstrate contribute to increased college enrollment and success. We find substantial positive effects of the incentive program on each of the incented behaviors: Treated students were more likely to engage regularly with a college advisor; apply to well-matched colleges and universities; and meet with an advisor to review their financial aid awards and discuss college costs. Yet students randomly offered the incentives were no more likely to enroll at higher-quality colleges and universities, despite being high in the distribution of college entrance exam scores and from a socioeconomic background that many institutions indicate is central to their diversity goals. Student responses to a survey administered the summer and fall after high school suggest that lack of admission to the most selective institutions, lack of affordability at selective institutions to which students were admitted, and student preferences to attend institutions closer to home explain the lack of enrollment effects.

More →

The improvement of low-performing school systems is one potential strategy for mitigating educational inequality. Some evidence suggests districtwide reform may be more effective than school-level change, but limited research examines district-level turnaround. There is also little scholarship examining the effects of turnaround reforms on outcomes beyond the first few years of implementation, on outcomes beyond test scores, or on the effectiveness of efforts to replicate district improvement successes beyond an initial reform context. We study these topics in Massachusetts, home to the Lawrence district representing a rare case of demonstrated improvements in the early years of state takeover and turnaround and where state leaders have since intervened in three other contexts as a result. We use statewide student-level administrative data (2006-07 to 2018-19) and event study methods to estimate medium-term reform impacts on test and non-test outcomes across four Massachusetts-based contexts: Lawrence, Holyoke, Springfield, and Southbridge. We find substantial district improvement was possible although sustaining the rate of gains was more complicated. Replicating gains in new contexts was also possible but not guaranteed. 

More →

Numerous studies have demonstrated a strong link between participation in pre-K programs and both short-term student achievement and positive later-life outcomes.  Existing evidence primarily stems from experimental studies of small-scale, high-quality programs conducted in the 1960s and 1970s and analyses of the federal Head Start program.  Meanwhile, evidence on state-funded pre-K programs, with no income restrictions, is scant and inconclusive.  Using enrollment lotteries for over-subscribed school-based sites in Georgia’s universal pre-K program, we analyze the impact of participation on elementary school outcomes.  Lottery winners enter kindergarten more prepared in both math and reading than non-winning peers. Gains fade by the end of kindergarten, and some negative achievement effects emerge by grade 4. Free-and-reduced-price meal (FRPM) students benefit more compared to non-FRPM students in later grades, suggesting greater benefits from attendance for disadvantaged students.  Although we find no effects for discipline, lottery winners had one fewer absence each grade after kindergarten.

More →

This study leverages national data and a quasi-experimental design to examine the influence of enrolling in an exclusively online degree program on students’ likelihood of completing their degree. We find that enrolling in an exclusively online degree program had a negative influence on students’ likelihood of completing their bachelor’s degree or any degree when compared to their otherwise-similar peers who enrolled in at least some face-to-face courses. The negative relationship between exclusively online enrollment and students’ likelihood of bachelor’s degree completion was relatively consistent among White, Black, Hispanic, Asian, low-income, and military students. Findings focused solely on those students enrolled in exclusively online degree programs revealed that the negative influence of exclusively online enrollment was exacerbated when the student attended a for-profit four-year institution.

More →

Although existing research suggests that students benefit on a range of outcomes when they enroll in early algebra classes, policy efforts that accelerate algebra enrollment for large numbers of students often have negative effects. Explanations for this apparent contradiction often emphasize the potential role of teacher and peer effects, which could create positive effects for individual students placed into early algebra that would not translate to larger-scale policies. We use detailed data from Oregon that contain information on the teachers and peers to whom students are exposed in order to investigate these explanations. Our regression discontinuity analyses replicate key findings from prior studies, indicating that placement in eighth-grade algebra boosts student achievement in math and English language arts. We then demonstrate that eighth-grade algebra placement positively affects the achievement level of students’ classmates, as well as the years of experience and value added of students’ math teachers. The effects on peer composition that we observe are large enough to plausibly explain the majority of the effects of eighth-grade algebra on student test scores.

More →

While multiple studies have examined the impact of school turnaround, less is known about reforms under the Every Student Succeeds Act (ESSA). To advance this literature, we examine North Carolina’s Restart (NCR) model. NCR aligns with ESSA by giving school leaders increased flexibility. Also, NCR differs from previous turnaround models by repackaging a traditionally sanction-based approach to instead motivate school leaders with increased autonomy. Using comparative interrupted time series models, we find positive NCR effects in math, but not in English Language Arts or on non-test-based student outcomes. Also, nearly a quarter of the positive NCR effect can be explained by decreased teacher and principal turnover. These results provide evidence to support current shifts toward reform models featuring local autonomy.

More →

While recent studies have demonstrated the potential of automated feedback to enhance teacher instruction in virtual settings, its efficacy in traditional classrooms remains unexplored. In collaboration with TeachFX, we conducted a pre-registered randomized controlled trial involving 523 Utah mathematics and science teachers to assess the impact of automated feedback in K-12 classrooms. This feedback targeted “focusing questions” – questions that probe students’ thinking by pressing for explanations and reflection. Our findings indicate that automated feedback increased teachers’ use of focusing questions by 20%. However, there was no discernible effect on other teaching practices. Qualitative interviews revealed mixed engagement with the automated feedback: some teachers noticed and appreciated the reflective insights from the feedback, while others had no knowledge of it. Teachers also expressed skepticism about the accuracy of feedback, concerns about data security, and/or noted that time constraints prevented their engagement with the feedback. Our findings highlight avenues for future work, including integrating this feedback into existing professional development activities to maximize its effect.

More →

Practice-based teacher education has increasingly been adopted as an alternative to more traditional, conceptually-focused pedagogies, yet the field lacks causal evidence regarding the relative efficacy of these approaches. To address this issue, we randomly assigned 185 college students to one of three experimental conditions reflective of common conceptually-focused and practice-based teacher preparation pedagogies. We find significant and large positive effects of practice-based pedagogies on participants’ skills in eliciting and responding to student thinking as demonstrated through a written assessment and a short teaching episode. Our findings contribute to a developing evidence base that can assist policymakers and teacher educators in designing effective teacher preparation at scale.

More →