- Benjamin L. Castleman
Search for EdWorkingPapers here by author, title, or keywords.
Benjamin L. Castleman
Colleges have increasingly turned to predictive analytics to target at-risk students for additional support. Most of the predictive analytic applications in higher education are proprietary, with private companies offering little transparency about their underlying models. We address this lack of transparency by systematically comparing two important dimensions: (1) different approaches to sample and variable construction and how these affect model accuracy; and (2) how the selection of predictive modeling approaches, ranging from methods many institutional researchers would be familiar with to more complex machine learning methods, impacts model performance and the stability of predicted scores. The relative ranking of students’ predicted probability of completing college varies substantially across modeling approaches. While we observe substantial gains in performance from models trained on a sample structured to represent the typical enrollment spells of students and with a robust set of predictors, we observe similar performance between the simplest and most complex models.
Growing experimental evidence demonstrates that low-touch informational, nudge, and virtual advising interventions are ineffective at improving postsecondary educational outcomes for economically-disadvantaged students at scale. Intensive in-person college advising programs are a considerably higher-touch and more resource intensive strategy; some programs provide students with dozen of hours of individualized assistance starting in high school and continuing through college, and can cost thousands of dollars per student served. Despite the magnitude of this investment, causal evidence on these programs' impact is quite limited, particularly for programs that serve Hispanic students, the fastest growing segment of U.S. college enrollees. We contribute new evidence on the impact of intensive college advising programs through a multi-cohort RCT of College Forward, which provides individualized advising from junior year of high school through college for a majority Hispanic student population in Texas. College Forward leads to a 7.5 percentage point increase in enrollment in college, driven entirely by increased enrollment at four-year universities. Students who receive College Forward advising are nearly 12 percentage points more likely to persist to their third year of college. While more costly and harder to scale than low-touch interventions, back of the envelope calculations suggest that the benefit from increased college graduation likely induced by the program outweighs operating costs in less than two years following college completion.
The COVID-19 pandemic led to an abrupt shift from in-person to virtual instruction in Spring 2020. Using a difference-in-differences framework that leverages within-course variation on whether students started their Spring 2020 courses in person or online, we estimate the impact of this shift on the academic performance of Virginia’s community college students. We find that the shift to virtual instruction resulted in a 6.7 percentage point decrease in course completion, driven by increases in both course withdrawal and failure. Faculty experience teaching a course online did not mitigate the negative effects of moving to virtual instruction.
The Post-9/11 GI Bill allows service members to transfer generous education benefits to a dependent. We run a large scale experiment that encourages service members to consider the transfer option among a population that includes individuals for whom the transfer benefits are clear and individuals for whom the net-benefits are significantly more ambiguous. We find no impact of a one-time email about benefits transfer among service members for whom we predict considerable ambiguity in the action, but sizeable impacts among service members for whom education benefits transfer is far less ambiguous. Our work contributes to the nascent literature investigating conditions when low-touch nudges at scale may be effective. JEL Classification: D15, D91, H52, I24
Tens of millions of Americans have lost their jobs in the wake of the COVID-19 health and economic crisis, and a sizable share of these job losses may be permanent. Unemployment rates are particularly high among adults without a college degree. Recent state policy efforts have focused on increasing re-enrollment and credentialing among adults with some college but no degree (SCND); these efforts are likely to accelerate given the COVID-19 disruptions to the U.S. economy. Yet little is actually known about the background characteristics, academic experiences, or labor market trajectories of this population. Using data from the Virginia Community College System (VCCS), we provide the first detailed profile on the academic, employment, and earnings trajectories of the SCND population, and how these compare on key measures to VCCS graduates. We also develop a framework for prioritizing which segments of the SCND population states might target for re-enrollment and completion interventions. This framework may be particularly useful to states that need to fill critical workforce shortages in healthcare and other sectors or re-train their workforce in the wake of mass unemployment and economic disruption stemming from the COVID-19 crisis.
In recognition of the complexity of the college and financial aid application process, and in response to insufficient access to family or school-based counseling among economically-disadvantaged populations, investments at the local, state, and federal level have expanded students’ access to college and financial aid advising. Experimental and quasi-experimental studies of these programs demonstrate that they can generate substantial improvements in the rate at which low-income students enroll and persist in college. While these programs are successful at the level of individual communities, the individualized, in-person college advising model faces numerous barriers to scale. In this paper, we report early results from an RCT of CollegePoint, an innovative, national college advising initiative that pursues a technology-enabled approach to provide students with sustained, intensive advising. Students assigned to CollegePoint are modestly more likely (1.5 percentage points, or 7.5 percent relative to the control) to enroll at the most selective colleges and universities (Barron’s 1 institutions), though we find no difference in enrollment patterns on other measures of college quality. We find suggestive evidence of variation in the impact of CollegePoint based on when students enrolled in the program. Students who enrolled in the spring of their junior year were 5.6 percentage points (22 percent relative to the control) more likely to enroll at one of the most selective colleges and universities in the country than students in the control group who also signed up in the spring of junior year but who were not assigned to the program.
Do nudge interventions that have generated positive impacts at a local level maintain efficacy when scaled state or nationwide? What specific mechanisms explain the positive impacts of promising smaller-scale nudges? We investigate, through two randomized controlled trials, the impact of a national and state-level campaign to encourage students to apply for financial aid for college. The campaigns collectively reached over 800,000 students, with multiple treatment arms to investigate different potential mechanisms. We find no impacts on financial aid receipt or college enrollment overall or for any student subgroups. We find no evidence that different approaches to message framing, delivery, or timing, or access to one-on-one advising affected campaign efficacy. We discuss why nudge strategies that work locally may be hard to scale effectively.
Student loan borrowing for higher education has emerged as a top policy concern. Policy makers at the institutional, state, and federal levels have pursued a variety of strategies to inform students about loan origination processes and how much a student has cumulatively borrowed, and to provide students with greater access to loan counseling. We conducted an experiment to evaluate the impact of an outreach campaign that prompted loan applicants at a large community college to make informed and active borrowing decisions and that offered them access to remote, one-onone assistance from a loan counselor. The intervention led students to reduce their unsubsidized loan borrowing by 7 percent, resulted in worse academic performance, and increased the likelihood of loan default during the three years after the intervention occurred. Our results suggest policy makers and higher education leaders should carefully examine the potential unintended consequences of efforts to reduce student borrowing, particularly in light of growing evidence regarding the counter-intuitive positive relationship between reduced borrowing levels and worse student academic and financial outcomes.
Despite broad public interest in Veterans' education, there is relatively little evidence documenting the postsecondary trajectories of military service members after they return to civilian life. In the current report we investigate how U.S. Army service member college enrollment and progression trends compare to a similar population of civilians, using Army administrative personnel data merged with administrative records from the National Student Clearinghouse and the Educational Longitudinal Study (ELS) of 2002. Civilians were nearly three times as likely to enroll in college within one year of high school graduation (or one year of separation). Civilians were also much more likely to earn a bachelor’s degree within the period of study than either of the Army samples. While members of minority race/ethnicity groups in both military samples enroll at higher rates than their white counterparts, racial/ethnic minorities do not graduate at higher rates than their white counterparts. We discuss policy implications of our analyses in the final section of our paper.