EdWorkingPapers

In a randomized trial that collects text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by human raters. An impact analysis can then be conducted to compare treatment and control groups, using the hand-coded scores as a measured outcome. This process is both time and labor-intensive, which creates a persistent barrier for large-scale assessments of text. Furthermore, enriching ones understanding of a found impact on text outcomes via secondary analyses can be difficult without additional scoring efforts. Machine-based text analytic and data mining tools offer one potential avenue to help facilitate research in this domain. For instance, we could augment a traditional impact analysis that examines a single human-coded outcome with a suite of automatically generated secondary outcomes. By analyzing impacts across a wide array of text-based features, we can then explore what an overall change signifies, in terms of how the text has evolved due to treatment. In this paper, we propose several different methods for supplementary analysis in this spirit. We then present a case study of using these methods to enrich an evaluation of a classroom intervention on young children’s writing. We argue that our rich array of findings move us from “it worked” to “it worked because” by revealing how observed improvements in writing were likely due, in part, to the students having learned to marshal evidence and speak with more authority. Relying exclusively on human scoring, by contrast, is a lost opportunity.

More →

High school Career and Technical Education (CTE) has received an increase in attention from both policymakers and researchers in recent years. This study fills a needed gap in the growing research base by examining heterogeneity within the wide range of programs falling under the broader CTE umbrella, and highlights the need for greater nuance in research and policy conversations that often consider CTE as monolithic. Examining multiple possible outcomes, including earnings, postsecondary education, and poverty avoidance, we find substantial differences in outcomes for students in fields as diverse as healthcare, IT, and construction. We also highlight heterogeneity for student populations historically overrepresented in CTE, and find large differences in outcomes for CTE students, particularly by gender.

More →

We examine the effects of disseminating academic performance data—either status, growth, or both—on parents’ school choices and their implications for racial, ethnic, and economic segregation. We conduct an online survey experiment featuring a nationally representative sample of parents and caretakers of children age 0-12. Participants choose between three randomly sampled elementary schools drawn from the same school district. Only growth information—alone and not in concert with status information—has clear and consistent desegregating consequences. Because states that include growth in their school accountability systems have generally done so as a supplement to and not a replacement for status, there is little reason to expect that this development will influence choice behavior in a manner that meaningfully reduces school segregation.

More →

Paraeducators perform multiple roles in U.S. classrooms, including among others preparing classroom activities, working with students individually and in small groups, supporting individualized programming for students with disabilities, managing classroom behavior, and engaging with parents and communities. Yet, little research provides insights into this key group of educators. This study combines an analysis of national administrative data to describe the paraeducator labor market with a systematic review of collective bargaining agreements and other job-defining documents in ten case-study districts. We find a large and expanding labor market of paraeducators, far more diverse along ethnic and racial lines than certified teachers but with far lower wages, fewer performance incentives, less professional development, and fewer opportunities for advancement within the profession.

More →

Adequately saving for retirement requires both planning and knowledge about available retirement savings options. Teachers participate in a complex set of different plan designs and benefit tiers, and many do not participate in Social Security. While teachers represent a large part of the public workforce, relatively little is known regarding their knowledge about and preparation for retirement. We administered a survey to a nationally representative sample of teachers through RAND’s American Teacher Panel and asked teachers about their retirement planning and their employer-sponsored retirement plans. We find that while most teachers are taking steps to prepare for retirement, many teachers lack the basic retirement knowledge necessary to plan effectively. Teachers struggled to identify their plan type, how much they are contributing to their plans, retirement eligibility ages, and who contributes to Social Security. These results suggest that teacher retirement reform may not be disruptive for teachers and that better, simpler, and clearer information about teacher retirement plans would be beneficial.

More →

Nearly half of students who enter college do not graduate. The majority of efforts to increase college completion have focused on supporting students before or soon after they enter college, yet many students drop out after making significant progress towards their degree. In this paper, we report results from a multi-year, large-scale experimental intervention conducted across five states and 20 broad-access, public colleges and universities to support students who are late in their college career but still at risk of not graduating. The intervention provided these “near-completer” students with personalized text messages that encouraged them to connect with campus-based academic and financial resources, reminded them of upcoming and important deadlines, and invited them to engage (via text) with campus-based advisors. We find little evidence that the message campaign affected academic performance or attainment in either the full sample or within individual higher education systems or student subgroups. The findings suggest low-cost nudge interventions may be insufficient for addressing barriers to completion among students who have made considerable academic progress.

More →

Between 1935-1940 the Home Owners' Loan Corporation (HOLC) assigned A (minimal risk) to D (hazardous) grades to neighborhoods that reflected their lending risk from previously issued loans and visualized these grades on color-coded maps, which arguably influenced banks and other mortgage lenders to provide or deny home loans within residential neighborhoods. In this study, we leverage a spatial analysis of 144 HOLC-graded core-based statistical areas (CBSAs) to understand how HOLC maps relate to current patterns of school and district funding, school racial diversity, and school performance. We find that schools and districts located today in historically redlined D neighborhoods have less district per-pupil total revenues, larger shares of Black and non-White student bodies, less diverse student populations, and worse average test scores relative to those located in A, B, and C neighborhoods. Conversely, at the school level, we find that per-pupil total expenditures are better for those schools operating in previously redlined D neighborhoods. Consequently, these schools also have the largest shares of low-income students. Our nationwide results are, on the whole, consistent by region and after controlling for CBSA. Finally, we document a persistence in these patterns across time, with overall positive time trends regardless of HOLC security rating but widening gaps between D vs. A, B, and C outcomes. These findings suggest that education policymakers need to consider the historical implications of redlining and past neighborhood inequality on neighborhoods today when designing modern interventions focused on improving the life outcomes of students of color and students from low-socioeconomic backgrounds.

More →

Personnel evaluation systems have historically failed to identify and remediate low-performing teachers. In 2012, Chicago Public Schools implemented an evaluation system that incorporated remediation and dismissal plans for low-rated teachers. Regression discontinuity estimates indicate that the evaluation reform increased the exit of low-rated tenured teachers by 50 percent. The teacher labor supply available to replace low-rated teachers was higher performing on multiple dimensions, and instrumental variables estimates indicate that policy-induced exit of low-rated teachers significantly improved teacher quality in subsequent years. Policy simulations show that the teacher labor supply in Chicago is sufficient to remove significantly more low-performing teachers.

More →

Human-capital formation in school depends largely on the selection and retention of teachers. I conduct a discrete-choice experiment with responses linked to administrative teacher and student records to examine teacher preferences for compensation structure and working conditions. I calculate willingness-to-pay for a rich set of work attributes. High-performing teachers have similar preferences to other teachers, but they have stronger preferences for performance pay. Taking the preference estimates at face value I explore how schools should structure compensation to meet various objectives. Under each objective, schools appear to underpay in salary and performance pay while overpaying in retirement. Restructuring compensation can increase both teacher welfare and student achievement.

More →

We assess whether a light-touch intervention can increase socioeconomic and racial diversity in undergraduate Economics. We randomly assigned over 2,200 students a message with basic information about the Economics major; the basic message combined with an emphasis on the rewarding careers or financial returns associated with the major; or no message. Messages increased the proportion of first generation or underrepresented minority (URM) students majoring in Economics by five percentage points. This effect size was sufficient to reverse the gap in Economics majors between first generation/URM students and students not in these groups. Effect sizes were larger and more precise for better-performing students and first generation students. Extrapolating to the full sample, the treatment would double the proportion of first generation and underrepresented minority students majoring in Economics.

More →