Search EdWorkingPapers

Search for EdWorkingPapers here by author, title, or keywords.

EdWorkingPapers

Lindsay C. Page, Jeonghyun Lee, Hunter Gehlbach.

Success in postsecondary education requires students to engage with their institution both academically and administratively. As with the transition to college, administrative requirements students face once enrolled can be substantial. Missteps with required processes can threaten students’ ability to persist. During the 2018-19 academic year, Georgia State University implemented an artificially intelligent text-based chatbot to provide proactive outreach and support to help undergraduates navigate administrative processes and take advantage of campus resources. A team of centralized university administrators orchestrated outreach “campaigns” to support students across three broad domains: (1) academic supports; (2) social and career supports; and (3) administrative processes. We investigate GSU’s implementation of this persistence-focused chatbot through an experimental study. Of the three message domains, outreach was most effective when focused on administrative processes, many of which were time-sensitive and for which outreach could be targeted specifically to students for whom it was relevant based on administrative data. In contrast, outreach to encourage take up of other supports had little effect on student behavior. By the end of the academic year, rates of FAFSA filing and registration for the subsequent fall semester were approximately three percentage points higher, suggesting positive effects on year-to-year college persistence. The positive effects on fall enrollment persisted into summer 2019, at which time the GSU administration judged that the study results were compelling enough to conclude the experiment and roll the chatbot system out to all students. We situate our findings in the literature on nudge-type efforts to support college access and success to draw lessons regarding their effective use.

More →


Francis A. Pearman, II.

This study examines the relationship between county-level estimates of implicit racial bias and black-white test score gaps in U.S. schools. Data from over 1 million respondents from across the United States who completed an online version of the Race Implicit Association Test (IAT) were combined with data from the Stanford Education Data Archive covering over 300 million test scores from U.S. schoolchildren in grades 3 through 8. Two key findings emerged. First, in both bivariate and multivariate models, counties with higher levels of racial bias had larger black-white test score disparities. The magnitude of these associations were on par with other widely accepted predictors of racial test score gaps, including racial gaps in family income and racial gaps in single parenthood. Second, the observed relationship between collective rates of racial bias and racial test score gaps was explained by the fact that counties with higher rates of racial bias had schools that were characterized by more racial segregation and larger racial gaps in gifted and talented assignment as well as special education placement. This pattern is consistent with a theoretical model in which aggregate rates of racial bias affect educational opportunity through sorting mechanisms that operate both within and beyond schools.

More →


Marcos A. Rangel, Ying Shi.

We study racial bias and the persistence of first impressions in the context of education. Teachers who begin their careers in classrooms with large black-white score gaps carry negative views into evaluations of future cohorts of black students. Our evidence is based on novel data on blind evaluations and non-blind public school teacher assessments of fourth and fifth graders in North Carolina. Negative first impressions lead teachers to be significantly less likely to over-rate but not more likely to under-rate black students’ math and reading skills relative to their white classmates. Teachers' perceptions are sensitive to the lowest-performing black students in early classrooms, but non-responsive to highest-performing ones. This is consistent with the operation of confirmatory biases. Since teacher expectations can shape grading patterns and sorting into academic tracks as well as students’ own beliefs and behaviors, these findings suggest that novice teacher initial experiences may contribute to the persistence of racial gaps in educational achievement and attainment.

More →


David M. Quinn.

A vast research literature documents racial bias in teachers’ evaluations of students.  Theory suggests bias may be larger on grading scales with vague or overly-general criteria versus scales with clearly-specified criteria, raising the possibility that well-designed grading policies may mitigate bias.  This study offers relevant evidence through a randomized web-based experiment with 1,549 teachers.  On a vague grade-level evaluation scale, teachers rated a student writing sample lower when it was randomly signaled to have a Black author, versus a White author.  However, there was no evidence of racial bias when teachers used a rubric with more clearly-defined evaluation criteria.  Contrary to expectation, I found no evidence that the magnitude of grading bias depends on teachers’ implicit or explicit racial attitudes.               

More →


Kelli A. Bird, Benjamin L. Castleman, Brett Fischer, Benjamin T. Skinner.

Tens of millions of Americans have lost their jobs in the wake of the COVID-19 health and economic crisis, and a sizable share of these job losses may be permanent. Unemployment rates are particularly high among adults without a college degree. Recent state policy efforts have focused on increasing re-enrollment and credentialing among adults with some college but no degree (SCND); these efforts are likely to accelerate given the COVID-19 disruptions to the U.S. economy. Yet little is actually known about the background characteristics, academic experiences, or labor market trajectories of this population. Using data from the Virginia Community College System (VCCS), we provide the first detailed profile on the academic, employment, and earnings trajectories of the SCND population, and how these compare on key measures to VCCS graduates. We also develop a framework for prioritizing which segments of the SCND population states might target for re-enrollment and completion interventions. This framework may be particularly useful to states that need to fill critical workforce shortages in healthcare and other sectors or re-train their workforce in the wake of mass unemployment and economic disruption stemming from the COVID-19 crisis.

More →


Jing Liu, Julie Cohen.

Valid and reliable measurements of teaching quality facilitate school-level decision-making and policies pertaining to teachers, but conventional classroom observations are costly, prone to rater bias, and hard to implement at scale. Using nearly 1,000 word-to-word transcriptions of 4th- and 5th-grade English language arts classes, we apply novel text-as-data methods to develop automated, objective measures of teaching to complement classroom observations. This approach is free of rater bias and enables the detection of three instructional factors that are well aligned with commonly used observation protocols: classroom management, interactive instruction, and teacher-centered instruction. The teacher-centered instruction factor is a consistent negative predictor of value-added scores, even after controlling for teachers’ average classroom observation scores. The interactive instruction factor predicts positive value-added scores.

More →


David M. Houston, Michael B. Henderson, Paul E. Peterson, Martin R. West.

States and districts are increasingly incorporating measures of achievement growth into their school accountability systems, but there is little research on how these changes affect the public’s perceptions of school quality. We conduct a nationally representative online survey experiment to identify the effects of providing participants with information about their local school districts’ average achievement status and/or average achievement growth. In the control group, participants who live in higher status districts tend to grade their local schools more favorably. The provision of status information does not fundamentally alter this relationship. The provision of growth information, however, reshapes Americans’ views about educational performance. Once informed, participants’ evaluations of their local public schools better reflect the variation in district growth.

More →


David M. Quinn.

The “achievement gap” has long dominated mainstream conversations about race and education.  Some scholars warn that the discourse around racial gaps perpetuates stereotypes and promotes the adoption of deficit-based explanations that fail to appreciate the role of structural inequities.  I investigate through three randomized experiments.  Results indicate that a TV news story about racial achievement gaps (versus a control or counter-stereotypical video) led viewers to express more exaggerated stereotypes of Black Americans as lacking education (study 1: ES=.30 SD; study 2: ES=.38 SD) and may have increased viewers’ implicit stereotyping of Black students as less competent than White students (study 1: ES=.22 SD; study 2: ES=.12 SD, n.s.).  The video did not affect viewers’ explicit competence-related racial stereotyping, the explanations they gave for achievement inequalities, or their prioritization of ending achievement inequalities.  After two weeks, the effect on stereotype exaggeration faded.  Future research should probe how we can most productively frame educational inequality by race.

More →


Angela Johnson, Megan Kuhfeld, James Soland.

Nearly one in five U.S. students attends a rural school, yet we know very little about achievement gaps and academic growth in rural schools. This study leverages a unique dataset that includes longitudinal test scores for more than five million 3rd to 8th grade students in approximately 17,000 public schools across the 50 states, including 900,000 students attending 4,727 rural schools. We find rural achievement and growth to be slightly above public schools. But there is considerable heterogeneity by student race/ethnicity. For all grades and subjects, White-Black and White-Hispanic gaps are smaller in rural schools than gaps nationwide, and White-Native American gaps are larger in rural schools than gaps nationwide. Separate analyses by racial/ethnic subgroup show that rural Black, Hispanic, and Native American students are often growing slower than their respective subgroup national average. In contrast, White students are often growing faster than the national average for White students.

More →


Lindsay C. Page, Matthew A. Lenard, Luke Keele.

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practice for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in Wake County, North Carolina.

More →