Search EdWorkingPapers

Search for EdWorkingPapers here by author, title, or keywords.

Program and policy effects

Matthew Kraft, Heather Hill.

This paper describes and evaluates a web-based coaching program designed to support teachers in implementing Common Core-aligned math instruction. Web-based coaching programs can be operated at relatively lower costs, are scalable, and make it more feasible to pair teachers with coaches who have expertise in their content area and grade level. Results from our randomized field trial document sizable and sustained effects on both teachers’ ability to analyze instruction and on their instructional practice, as measured the Mathematical Quality of Instruction (MQI) instrument and student surveys. However, these improvements in instruction did not result in corresponding increases in math test scores as measured by state standardized tests or interim assessments. We discuss several possible explanations for this pattern of results.

More →

Jade M. Jenkins, C. Kevin Fortner.

We provide causal estimates of the effects of delayed kindergarten entry on achievement outcomes by exploiting a policy change in the birthdate enrollment cutoff in North Carolina that forced children born in a six-week window to redshirt. Using multiple peer group comparisons, we identify impacts on achievement and gifted or disability identifications in third through fifth grades. Delayed entry provides small benefits to students’ math and reading achievement, and reduced identification of a disability; these impacts operate through cohort position and age advantages, and not from hold-out year experiences. Redshirting differentially benefitted low-income students, but further disadvantaged non-white students.


More →

Matthew Kraft.

Researchers commonly interpret effect sizes by applying benchmarks proposed by Cohen over a half century ago. However, effects that are small by Cohen’s standards are large relative to the impacts of most field-based interventions. These benchmarks also fail to consider important differences in study features, program costs, and scalability. In this paper, I present five broad guidelines for interpreting effect sizes that are applicable across the social sciences. I then propose a more structured schema with new empirical benchmarks for interpreting a specific class of studies: causal research on education interventions with standardized achievement outcomes. Together, these tools provide a practical approach for incorporating study features, cost, and scalability into the process of interpreting the policy importance of effect sizes.

More →

Download PDF808.62 KB

Tolani Britton, Birunda Chelliah, Millie Symns, Vandeen Campbell.

Research suggests that earning college credits in high school increases the likelihood of postsecondary progress and graduation. In this study, we measure the impact of dual enrollment in high school and college courses through the College Now (CN) program on college enrollment for students in New York City. We use a regression discontinuity design (RDD) that estimates the causal local average effect of the treatment — eligibility for dual enrollment in college classes while in high school — on college enrollment. We find that being eligible for CN leads to a 7% point increase in the likelihood of college enrollment and an 8.6% point increase in the likelihood of enrollment in a four-year college. Students who were eligible for CN and enrolled in CN were 20% points more likely to enroll in college.

More →

Patrick J. Wolf, John F. Witte, Brian Kisida.

The Milwaukee Parental Choice Program (MPCP), the first modern private school choice program in the United States, has grown from 341 students attending 7 private schools in 1990 to 27,857 students attending 126 private schools in 2019. The MPCP has been subject to extensive study focused largely on student performance on standardized tests. This study presents new data on the college enrollment, persistence, and graduation of MPCP and MPS students who were tracked over 12 years beginning in 2006. MPCP participants are compared with a matched sample of MPS students who lived in the same neighborhood and had similar demographic characteristics and test scores at the beginning of the study. The collective evidence in this paper indicates that students in the MPCP program have greater educational attainment than the comparison group, as measured by college experience and outcomes. Most of the college attainment benefits of the MPCP are clear for both students who were in ninth grade at the beginning of the study, for whom positive attainment effects have previously been reported, and students who were initially enrolled in grades three through eight, who we examine here for the first time. As of 2018, MPCP students have spent more total years in a four-year college than their MPS peers. The MPCP students in the grade three through eight sample attained college degrees at rates that are statistically significantly higher than their matched MPS peers.

More →

Eric Brunner, Shaun Dougherty, Stephen Ross .

We examine the effect of admission to 16 stand-alone technical high schools within the Connecticut Technical High School System (CTHSS) on student educational and labor market outcomes. To identify the causal effect of admission on student outcomes, we exploit the fact that CTHSS utilizes a score-based admissions system and identify the effect of admission using a regression discontinuity approach. We find that male students attending one of the technical high schools are approximately 10 percentage points more likely to graduate from high school and 8 percentage points less likely to attend college, although there is some evidence that the negative effects on college attendance fade over time. We also find that male students attending a technical high school have quarterly earnings that are approximately 31% higher. Analyses of potential mechanisms behind these results reveal that male students that attend a technical high school have higher 9th grade attendance rates and higher 10th grade test scores. We find little evidence that attending a technical high school affects the educational or labor outcomes of women. These effects appear relatively broad based across different types of students in that we find little evidence of heterogeneity in these effects over student attributes like race and ethnicity, free lunch eligibility or residence in a poor, central city school district. However, when distinguishing between students based on the Career and Technical Education (CTE) offerings of the high school that these students likely would have attended, we find that the effects of admission to a CTHSS school are noticeably larger when the counterfactual high school has less CTE offerings.   

More →

Elizabeth Setren, Kyle Greenberg, Oliver Moore, Michael Yankovich.

In a flipped classroom, an increasingly popular pedagogical model, students view a video lecture at home and work on exercises with the instructor during class time. Advocates of the flipped classroom claim the practice not only improves student achievement, but also ameliorates the achievement gap. We conduct a randomized controlled trial at West Point and find that the flipped classroom produced short term gains in Math and no effect in Economics, but that the flipped model broadened the achievement gap: effects are driven by white, male, and higher achieving students. We find no long term average effects on student learning, but the widened achievement gap persists. Our findings demonstrate feasibility for the flipped classroom to induce short term gains in student learning; however, the exacerbation of the achievement gap, the effect fade-out, and the null effects in Economics suggest that educators should exercise caution when considering the model.

More →

Zack Mabel, CJ Libassi, Michael Hurwitz.

Policymakers are increasingly including early-career earnings data in consumer-facing college search tools to help students and families make more informed post-secondary education decisions. We offer new evidence on the degree to which existing college-specific earnings data equips consumers with useful information by documenting the level of selection bias in the earnings metrics reported in the U.S. Department of Education’s College Scorecard. Given growing interest in reporting earnings by college and major, we focus on the degree to which earnings differences across four-year colleges and universities can be explained by differences in major composition across institutions. We estimate that more than three-quarters of the variation in median earnings across institutions is explained by observable factors, and accounting for differences in major composition explains over 30 percent of the residual variation in earnings after controlling for institutional selectivity, student composition, and local cost of living differences. We also identify large variations in the distribution of earnings within colleges; as a result, comparisons of early-career earnings can be extremely sensitive to whether the median, 25th, or 75th percentiles are presented. Taken together, our findings indicate that consumers can easily draw misleading conclusions about institutional quality when using publicly available earnings data to compare institutions.

More →

Elizabeth Setren.

Using novel variation in special education and English Language Learner classification from admissions lotteries, I find that students can achieve large academic gains without specialized services. Enrolling in a Boston charter school doubles the likelihood that students lose their special education or English Language Learner status, but exposes students to a high-performing general education program that includes high intensity tutoring, data driven instruction, and increased instructional time. The positive effects extend to college: charters nearly double the likelihood that English Language Learners enroll in four-year colleges and quadruple the likelihood that special education students graduate from two-year college. A multiple instrument strategy suggests that high quality general education practices drive the gains and finds no detrimental effect from lower classification rates.

More →

Anjali Adukia.

Programs that provide lower-skill employment are a popular anti-poverty strategy in developing countries, with India's employment-guarantee program (MGNREGA) employing adults in 23% of Indian households.  A potential concern is that guaranteeing lower-skill employment opportunities may discourage investment in human capital and long-run income growth.  Using large-scale administrative data and household survey data, I estimate precise spillover impacts on education that reject substantive declines in children's education from the government's rollout of MGNREGA.  I estimate that these small negative impacts are inexpensive to counteract, particularly compared to MGNREGA expenditures on rural employment and poverty alleviation.

More →