Search for EdWorkingPapers here by author, title, or keywords.
Researchers are rarely satisfied to learn only whether an intervention works, they also want to understand why and under what circumstances interventions produce their intended effects. These questions have led to increasing calls for implementation research to be included in high quality studies with strong causal claims. Of critical importance is determining whether an intervention can be delivered with adherence to a standardized protocol, and the extent to which an intervention protocol can be replicated across sessions, sites, and studies. When an intervention protocol is highly standardized and delivered through verbal interactions with participants, a set of natural language processing (NLP) techniques termed semantic similarity can be used to provide quantitative summary measures of how closely intervention sessions adhere to a standardized protocol, as well as how consistently the protocol is replicated across sessions. Given the intense methodological, budgetary and logistical challenges for conducting implementation research, semantic similarity approaches have the benefit of being low-cost, scalable, and context agnostic for use. In this paper, we demonstrate how semantic similarity approaches may be utilized in an experimental evaluation of a coaching protocol on teacher pedagogical skills in a simulated classroom environment. We discuss strengths and limitations of the approach, and the most appropriate contexts for applying this method.
Most public colleges and universities rely heavily on state financial support. As state budgets have tightened in recent decades, appropriations for higher education have declined substantially. Despite concerns expressed by policymakers and scholars that the declines in state support have reduced the return to education investment for public sector students, little evidence exists that can identify the causal effect of these funds on long-run outcomes. We present the first such analysis in the literature using new data that leverages the merger of two rich datasets: consumer credit records from the New York Fed's Consumer Credit Panel (CCP), sourced from Equifax, and administrative college enrollment and attainment data from the National Student Clearinghouse. We overcome identification concerns related to the endogeneity of state appropriation variation using an instrument that interacts the baseline share of total revenue that comes from state appropriations at each public institution with yearly variation in state-level appropriations. Our analysis is conducted separately for two-year and four-year students, and we analyze individuals into their mid-30s. For four-year students, we find that state appropriation increases lead to substantially lower student debt originations. They also react to appropriation increases by shortening their time to degree, but we find little effect on other outcomes. In the two-year sector, state appropriation increases lead to more collegiate and post-collegiate educational attainment, more educational debt consistent with the increased educational attainment, but lower likelihood of delinquency and default. State support also leads to more car and home ownership with lower adverse debt outcomes, and these students experience substantial increases in their credit score and in the affluence of the neighborhood in which they live. Examining mechanisms, we find state appropriations are passed on to students in the form of lower tuition in the four-year sector with no institutional spending response. For community colleges, we find evidence of both price and quality mechanisms, the latter captured in higher educational resources in key spending categories. These results are consistent with the different pattern of effects we document in the four-year and two-year sectors. Our results underscore the importance of state support for higher education in driving student debt outcomes and the long-run returns to postsecondary investments that students experience.
We replicate and extend prior work on Florida’s Bright Futures merit aid scholarship to consider its effect on college enrollment and degree completion. We estimate causal impacts using a regression discontinuity design to exploit SAT thresholds that strongly determine eligibility. We find no positive impacts on attendance or attainment, and instrumental variable results generally reject estimates as small as 1-2 percentage points. Across subgroups, we do find that eligibility slightly reduces six-year associate degree attainment for lower-SES students and may induce small enrollment shifts among Hispanic and White students. Our findings of these minimal-at-best impacts contrast those of prior works, attributable in part to methodological improvements and more robust data, and further underscore the importance of study replication. (JEL: H75, I21, I22, I23, I28)
Nationally, 15% of first-time community college students were high school dual enrollment (DE) students, which raises concerns about how high school peers might influence college enrollees. Using administrative data from a large state community college system, we examine whether being exposed to a higher percentage of DE peers in entry-level (gateway) math and English courses influences non-DE enrollees’ performance. Using a two-way fixed effects model, our results indicate that college enrollees exposed to a higher proportion of DE peers had lower pass rates and grades in gateway courses, and higher course repetition rates. Supplemental student-level analysis suggests that greater exposure to DE peers during a student’s initial semester in college reduces next-term college persistence.
The Post-9/11 GI Bill allows service members to transfer generous education benefits to a dependent. We run a large scale experiment that encourages service members to consider the transfer option among a population that includes individuals for whom the transfer benefits are clear and individuals for whom the net-benefits are significantly more ambiguous. We find no impact of a one-time email about benefits transfer among service members for whom we predict considerable ambiguity in the action, but sizeable impacts among service members for whom education benefits transfer is far less ambiguous. Our work contributes to the nascent literature investigating conditions when low-touch nudges at scale may be effective. JEL Classification: D15, D91, H52, I24
Recent immigration policies have created massive uncertainty for international students to obtain F-1 visas. Yet, before the COVID-19 pandemic, student visa applicants already faced an approximately 27 percent refusal rate that varies by time and region. Using data on the universe of SAT takers between 2004 and 2015 matched with college enrollment records, we examine how the anticipated F-1 visa restrictiveness influences US undergraduate enrollment outcomes of international students. Using an instrumental variables approach, we find that a higher anticipated F-1 student visa refusal rate decreases the number of international SAT takers, decreases the probability of sending SAT scores to US colleges, and decreases international student enrollment in the US. The decreases are larger among international students with higher measured academic achievement. We also document academic achievement of international students and show that over 40 percent of high-scoring international SAT takers do not pursue US college education.
While current debates center on whether and how to admit immigrants to the United States, little attention has been paid to interventions designed to help immigrants integrate after they arrive. Public adult education programs are the primary policy lever for building the language skills of the over 23 million adults with limited English proficiency in the United States. We leverage the enrollment lottery of a publicly-funded adult English for Speakers of Other Languages (ESOL) program in Massachusetts to estimate the effects of English language training on voting behavior and employer-reported earnings. Attending ESOL classes more than doubles rates of voter registration and increases annual earnings by $2,400 (56%). We estimate that increased tax revenue from earnings gains fully pay for program costs over time, generating a 6% annual return for taxpayers. Our results demonstrate the social value of post-migration investments in the human capital of adult immigrants.
This paper estimates the relationship between students’ psychosocial and academic outcomes during their first three years enrolled at public, four-year institutions. Our sample is comprised of students from low-income backgrounds who applied for a competitive scholarship and enrolled at a four-year public institution. We follow two cohorts of entering students throughout their first three years on campus. We observe their cumulative GPA and persistence decisions each semester, and have annual measures of four psychosocial outcomes: mattering to campus, sense of belonging to campus, academic self-efficacy, and social self-efficacy. We find that psychosocial outcomes are moderately predictive of academic outcomes, with sense of belonging and academic self-efficacy emerging as most predictive of both cumulative GPA and persistence.
We examine the impact of the Thompson Scholars Learning Community (TSLC), a comprehensive college transition program serving students with a variety of majors, on students’ science, technology, engineering, and math (STEM)-related outcomes. We use an explanatory mixed-methods design, which prioritizes the quantitative analyses and uses qualitative analyses to contextualize and explain our quantitative findings. Overall, participating in TSLC does not make students more likely to declare a STEM major, although we do find a positive effect for students of color. TSLC students earn higher overall GPAs than their scholarship-only peers, and TSLC students majoring in STEM outperform scholarship-only STEM majors in STEM courses. Qualitative analyses suggest these results stem from the student-centered and proactive support the program provides students. Our results suggest that a disciplinarily-agnostic program can support student success in STEM, and may increase equitable representation in STEM fields.
Verification is a federally mandated process that requires selected students to further attest that the information reported on their FAFSA is accurate and complete. In this brief, we estimate institutional costs of administrating the FAFSA verification mandate and consider variation in costs by institution type and sector. Using data from 2014, we estimate that compliance costs to institutions in that year totaled nearly $500 million with the burden falling disproportionately on public institutions and community colleges, in particular. Specifically, we estimate that 22% of an average community college’s financial aid office operating budget is devoted to verification procedures, compared to 15% at public four-year institutions. Our analysis is timely, given that rates of FAFSA verification have increased in recent years.