- Beth E. Schueler
Search for EdWorkingPapers here by author, title, or keywords.
Beth E. Schueler
A significant share of education and development research uses data collected by workers called “enumerators.” It is well-documented that “enumerator effects”—or inconsistent practices between the individual people who administer measurement tools— can be a key source of error in survey data collection. However, it is less understood whether this is a problem for academic assessments or performance tasks. We leverage a remote phone-based mathematics assessment of primary school students and survey of their parents in Kenya. Enumerators were randomized to students to study the presence of enumerator effects. We find that both the academic assessment and survey was prone to enumerator effects and use simulation to show that these effects were large enough to lead to spurious results at a troubling rate in the context of impact evaluation. We therefore recommend assessment administrators randomize enumerators at the student level and focus on training enumerators to minimize bias.
School closures induced by COVID-19 placed heightened emphasis on alternative ways to measure student learning besides in-person exams. We leverage the administration of phone-based assessments (PBAs) measuring numeracy and literacy for primary school children in Kenya, along with in-person standardized tests administered to the same students prior to school shutdowns, to assess the validity of PBAs. Compared to repeated in-person assessments, PBAs did not severely misclassify students’ relative performance, but PBA scores did tend to be further from baseline in-person scores than repeated in-person assessments from each other. As such, PBAs performed well at measuring aggregate but not individual learning levels. Administrators can therefore use these tools for aggregate measurement, such as in the context of impact evaluation, but be wary of PBAs for individual-level tracking or high-stakes decisions. Results also reveal the importance of making deliberate efforts to reach a representative sample and selecting items that provide discriminating power.
Covid-19-induced school closures generated great interest in tutoring as a strategy to make up for lost learning time. Tutoring is backed by a rigorous body of research, but it is unclear whether it can be delivered effectively remotely. We study the effect of teacher-student phone call interventions in Kenya when schools were closed. Schools (n=105) were randomly assigned for their 3rd, 5th and 6th graders (n=8,319) to receive one of two versions of a 7-week weekly math-focused intervention—5-minute accountability checks or 15-minute mini-tutoring sessions—or to the control group. Although calls increased student perceptions that teachers cared, accountability checks had no effect on math performance up to four months after the intervention and tutoring decreased math achievement among students who returned to their schools after reopening. This was, in part, because the relatively low-achieving students most likely to benefit from calls were least likely to return and take in-person assessments. Tutoring substituted away from more productive uses of time, at least among returning students. Neither intervention affected enrollment. Tutoring remains a valuable tool but to avoid unintended consequences, careful attention should be paid to aligning tutoring interventions with best practices and targeting interventions to those who will benefit most.
Local school boards have primary authority for running educational systems in the U.S. but little is known empirically about the merits of this arrangement. State takeovers of struggling districts represent a rare alternative form of educational governance and have become an increasingly common response to low performance. However, limited research explores whether this effectively improves student outcomes. We track all takeovers nationwide from the late 1980s, when the first takeovers occurred, through 2016 and describe takeover districts. While these districts are low performing, we find academic performance plays less of a role in predicting takeover for districts serving larger concentrations of African American students. We then use a new data source allowing for cross-state comparisons of student outcomes to estimate the effect of takeovers that occurred between 2011 and 2016. On average, we find no evidence that takeover generates academic benefits. Takeover appears to be disruptive in the early years of takeover, particularly to English Language Arts achievement, although the longer-term effects are less clear. We also observe considerable heterogeneity of effects across districts. Takeovers were least effective in districts with higher baseline achievement and least harmful in majority Latinx communities. Leaders should be cautious about using takeover without considering local context and a better understanding of why some takeovers are more effective than others.
The public narrative surrounding efforts to improve low-performing K-12 schools in the U.S. has been notably gloomy. Observers argue that either nothing works or we don’t know what works. At the same time, the federal government is asking localities to implement evidence-based interventions. But what is known empirically about whether school improvement works, how long it takes, which policies are most effective, and which contexts respond best to intervention? We meta-analyze 141 estimates from 67 studies of turnaround policies implemented post-NCLB. On average, these policies have had a moderate positive effect on math but no effect on ELA achievement as measured by high-stakes exams. We find evidence of positive impacts on low-stakes exams in STEM and humanities subjects and no evidence of harm on non-test outcomes. Some elements of reform, namely extended learning time and teacher replacements, predict greater effects. Contexts serving majority-Latinx populations have seen the largest improvements.