Search EdWorkingPapers

Search EdWorkingPapers by author, title, or keywords.

Program and policy effects

Noam Angrist, Rachael Meager.

Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. We study this variation by aggregating evidence from prior randomized trials across five contexts, and use the results to inform a new randomized trial. We find two factors explain most of the heterogeneity in effects across contexts: the degree of implementation (intention-to-treat or treatment-on-the-treated) and program delivery model (teachers or volunteers). Accounting for these implementation factors yields high generalizability, with similar effect sizes across studies. Thus, reporting treatment-on-the-treated effects, a practice which remains limited, can enhance external validity. We also introduce a new Bayesian framework to formally incorporate implementation metrics into evidence aggregation. Results show targeted instruction delivers average learning gains of 0.42 SD when taken up and 0.85 SD when implemented with high fidelity. To investigate how implementation can be improved in future settings, we run a new randomized trial of a targeted instruction program in Botswana. Results demonstrate that implementation can be improved in the context of a scaling program with large causal effects on learning. While research on implementation has been limited to date, our findings and framework reveal its importance for impact evaluation and generalizability.

More →


Brian McManus, Jessica Howell, Michael Hurwitz.

The impact of test-optional college admissions policies depends on whether applicants act strategically in disclosing test scores. We analyze individual applicants’ standardized test scores and disclosure behavior to 50 major US colleges for entry in fall 2021, when Covid-19 prompted widespread adoption of test-optional policies. Applicants withheld low scores and disclosed high scores, including seeking admissions advantages by conditioning their disclosure choices on their other academic characteristics, colleges’ selectivity and testing policy statements, and the Covid-related test access challenges of the applicants’ local peers. We find only modest differences in test disclosure strategies by applicants’ race and socioeconomic characteristics.

More →


Shaun M. Dougherty, Mary M. Smith, Beth Kelly .

Prior research has clearly established the substantial expected payoffs to investments in early childhood education. However, the ability to deliver early childhood programs differs across communities with access to high quality programing especially hard to establish in rural communities. We study one program, Early Steps to School Success, to understand whether the provision of home visiting and book exchange programs in rural Kentucky can influence kindergarten readiness. Linking program data with the state longitudinal data system in Kentucky we create multiple comparison groups by matching children on known program qualification indicators to estimate whether Early Steps program participation was related to school readiness. Our estimates suggest that program participation resulted in small improvements to children’s kindergarten readiness, as measured by the Brigance kindergarten readiness assessment overall score and sub-scores in language, cognitive, and physical development. Results are not sensitive to our choice of comparison group, though they appear driven by the experiences of children who participate from birth through age five or from ages three-to-five only. Our findings suggest that the Early Steps home visiting intervention may be a worthwhile intervention for improving kindergarten preparedness for children living in rural contexts.

More →


Riley Acton, Jo Al Khafaji-King, Austin Smith.

We examine the impact of local labor market shocks and state unemployment insurance (UI) policies on student discipline in U.S. public schools. Analyzing school-level discipline data and firm-level layoffs in 23 states, we find that layoffs have little effect on discipline rates overall. However, effects differ across the UI benefit distribution. At the lowest benefit level ($265/week), a mass layoff increases out-of-school suspensions by 4.5%, with effects dissipating as UI benefits increase. Effects are consistently largest for Black students - especially in predominantly White schools - resulting in increased racial disproportionality in school discipline following layoffs in low-UI states.

More →


Will Davis, Daniel Kreisman, Tareena Musaddiq .

We estimate the effect of universal free school meal access through the Community Eligibility Program (CEP) on child BMI. Through the CEP, schools with high percentages of students qualified for free or reduced-priced meals can offer freebreakfast and lunchto all students. With administrative data from a large school district in Georgia, we use student-level BMI measures from the FitnessGram to compare within-student outcomes before and after CEP implementation across eligible and non-eligible schools. We find one year of CEP exposure increased expected BMI percentile by about 0.085 standard deviations, equivalent to a nearly 1.88- pound weight increase for a student of average height. We also find that the program led to a small increase in the likelihood of overweight and limited evidence of a small decrease in the likelihood of underweight. We do not find that the program increased student obesity risk. Examining the effects of CEP on child BMI by grade suggests that the overall effect is largely driven by students in middle schools, highlighting potential heterogeneity in the program’s impact across grades. The findings of this paper are relevant for researchers and policymakers concerned with the effects of universal free school meals on student health.

More →


Oded Gurantz, Yung-Yu Tsai.
Government programs impose eligibility requirements to balance the goals of improving welfare while minimizing waste. We study the impact of eligibility monitoring in the context of Federal Application for Federal Student Aid (FAFSA) submissions, where students may be subject to “verification” requirements that require them to confirm the accuracy of the data. Using a matching on observables design we do not find that students flagged for verification are less likely to enroll in college, which contrasts prior research. Verification reduces grant aid received but average changes are small, raising questions about the benefits of this administrative process.

More →


Taylor Odle, Jennifer Delaney.

Complexity and uncertainty in the college application process contribute to longstanding racial and socioeconomic disparities in enrollment. We leverage a large-scale experiment that combines an early guarantee of college admission with a proactive nudge, fee waiver, and structural application simplification to test the impacts of emerging “direct admissions” policies on students’ college-going behaviors. Students in the intervention were 2.7 percentage points (or 12%) more likely to submit a college application, with larger impacts for racially minoritized, first-generation, and low-income students. Students were most responsive to automatic offers from larger, higher quality institutions on the application margin, but were not more likely to subsequently enroll. In the face of growing adoption, we show this low-cost, low-touch intervention can move the needle on important college-going behaviors but is insufficient alone to increase enrollment given other barriers to access, including the ability to pay for college.

More →


Lindsay C. Page, Katharine Meyer, Jeonghyun Lee, Hunter Gehlbach.

College success requires students to engage with their institution academically and administratively. Missteps with required administrative processes can threaten student persistence and success. Through two experimental studies, we assessed the effectiveness of an artificially intelligent text-based chatbot that provided proactive outreach and support to college students to navigate administrative processes and use campus resources. In both the two-year and four-year college context, outreach was most effective when focused on discrete administrative processes such as filing financial aid forms or registering for courses which were acute, time-sensitive, and for which outreach could be targeted to those for whom it was relevant. In the context of replicating studies to better inform policy and programmatic decision making, we draw three core lessons regarding the effective use of nudge-type efforts to promote college success.

More →


Christopher Rick.

Congestion is a persistent and expensive problem, costing the nation collectively over $300 billion each year. Cities have generally attempted to address congestion using an unoriginal set of expensive strategies, like building new roads or expanding public transit, and many cities are considering implementing congestion pricing. Expanding school bus service may be a palatable solution because it provides a service instead of involving lengthy and costly construction or charging a new fee. School travel is also a sizeable portion of total daily tra c. Indeed, over 50 million children travel to and from school each day and their commutes account for about one-quarter of total daily commuter trips. School travel and school-provided transportation is generally the domain of school districts and not city governments and the school districts in most large cities are independent from city governments. This may lead to a coordination problem if school districts ignore congestion caused, or exacerbated by, school travel. To determine whether pupil transportation reduces congestion, I exploit the interaction of pupil transportation provision (variation in pupil transportation spending and school bus use within districts) and idiosyncratic, within-city and within-month variation in the percentage of weekdays that are instructional school days in a month. I build a rich, monthly, longitudinal data set for congestion, school days, and transportation policy for 51 cities from 2013 to 2019 and find congestion is significantly higher on school days and pupil transportation alleviates congestion caused by school children’s travel. A back-of-the-envelope calculation suggests cities should subsidize the additional spending needed by the school district to transport more students and lower congestion.

More →


Beth E. Schueler, Katherine E. Larned.

Few interventions reduce inequality in reading achievement, let alone higher order thinking skills, among adolescents. We study “policy debate”—an extracurricular activity focused on improving middle and high schoolers’ critical thinking, argumentation, and policy analysis skills—in Boston schools serving large concentrations of economically-disadvantaged students of color. Student fixed effects estimates show debate had positive impacts on ELA test scores of 0.13 SD, equivalent to 68% of a full year of average 9th grade learning. Gains were concentrated on analytical more than rote subskills. We find no harm to math, attendance, or disciplinary records, and evidence of positive effects on high school graduation and postsecondary enrollment. Impacts were largest among students who were lowest achieving prior to joining debate.

More →