Search EdWorkingPapers

Search for EdWorkingPapers here by author, title, or keywords.

Methodology, measurement and data

Jing Liu, Megan Kuhfeld, Monica Lee, Danett Song.

Noncognitive constructs such as self-efficacy, social awareness, and academic engagement are widely acknowledged as critical components of human capital, but systematic data collection on such skills in school systems is complicated by conceptual ambiguities, measurement challenges and resource constraints. This study addresses this issue by comparing the predictive validity of two most widely used metrics on noncogntive outcomes|observable academic behaviors (e.g., absenteeism, suspensions) and student self-reported social and emotional learning (SEL) skills|for the likelihood of high school graduation and postsecondary attainment. Our findings suggest that conditional on student demographics and achievement, academic behaviors are several-fold more predictive than SEL skills for all long-run outcomes, and adding SEL skills to a model with academic behaviors improves the model's predictive power minimally. In addition, academic behaviors are particularly strong predictors for low-achieving students' long-run outcomes. Part-day absenteeism (as a result of class skipping) is the largest driver behind the strong predictive power of academic behaviors. Developing more nuanced behavioral measures in existing administrative data systems might be a fruitful strategy for schools whose intended goal centers on predicting students' educational attainment.

More →


Alex Eble, Maya Escueta.

How much does family demand matter for child learning in contexts of extreme poverty? In rural Gambia, families with high aspirations for their children's future education and career, measured before children start school, go on to invest substantially more than other families in their children’s education. Despite this, essentially no children are literate or numerate three years later. When villages receive a highly-impactful, teacher-focused supply-side intervention, however, children of these families are 25 percent more likely to achieve literacy and numeracy than other children in the same village. Furthermore, improved supply enables these children to acquire other higher-level skills necessary for later learning and child development. In such settings, greater demand can map onto developmentally meaningful learning differences, but only with adequate complementary inputs.

More →


Kelli A. Bird, Benjamin L. Castleman, Yifeng Song, Renzhe Yu.

Data science applications are increasingly entwined in students’ educational experiences. One prominent application of data science in education is to predict students’ risk of failing a course in or dropping out from college. There is growing interest among higher education researchers and administrators in whether learning management system (LMS) data, which capture very detailed information on students’ engagement in and performance on course activities, can improve model performance. We systematically evaluate whether incorporating LMS data into course performance prediction models improves model performance. We conduct this analysis within an entire state community college system. Among students with prior academic history in college, administrative data-only models substantially outperform LMS data-only models and are quite accurate at predicting whether students will struggle in a course. Among first-time students, LMS data-only models outperform administrative data-only models. We achieve the highest performance for first-time students with models that include data from both sources. We also show that models achieve similar performance with a small and judiciously selected set of predictors; models trained on system-wide data achieve similar performance as models trained on individual courses.

More →


Derek Rury.

A student's class rank has important short and long-term effects on important educational outcomes. Despite our growing understanding of these rank effects, we still do not know how early in a child's academic career they begin. To address this, I use data from the Tennessee STAR project, which randomly assigned over 6,323 kindergarteners to classroom environments, to study the impact of kindergarten class rank on a host of short and long-run outcomes. I find a strong, causal relationship between one's kindergarten classroom rank and subsequent test scores, high school achievement and performance on college entrance exams. I also find that having a higher rank in kindergarten causes an increase in study effort, value of school and initiative in the classroom. I also leverage the design of project STAR to test various mechanisms and address several outstanding issues in the rank literature, including the role of tracking, parental effort and teacher-level characteristics in driving the effects of class rank.

More →


Shaun M. Dougherty, Mary M. Smith.

Career and technical education (CTE) has existed in the United States for over a century, and only in recent years have there been opportunities to assess the causal impact of participating in these programs while in high school. To date, no work has assessed whether the relative costs of these programs meet or exceed the benefits as described in recent evaluations. In this paper, we use available cost data to compare average costs per pupil in standalone high school CTE programs in Connecticut and Massachusetts to the most likely counterfactual schools. Under a variety of conservative assumptions about the monetary value of known educational and social benefits, we find that programs in Massachusetts offer clear positive returns on investment, whereas programs in Connecticut offer smaller, though mostly non-negative expected returns. We also consider the potential cost effectiveness of CTE programs offered in other contexts to address questions of generalizability.  

More →


Kate Antonovics, Sandra E. Black, Julie Berry Cullen, Akiva Yonah Meiselman.

Schools often track students to classes based on ability. Proponents of tracking argue it is a low-cost tool to improve learning since instruction is more effective when students are more homogeneous, while opponents argue it exacerbates initial differences in opportunities without strong evidence of efficacy. In fact, little is known about the pervasiveness or determinants of ability tracking in the US. To fill this gap, we use detailed administrative data from Texas to estimate the extent of tracking within schools for grades 4 through 8 over the years 2011-2019. We find substantial tracking; tracking within schools overwhelms any sorting by ability that takes place across schools. The most important determinant of tracking is heterogeneity in student ability, and schools operationalize tracking through the classification of students into categories such as gifted and disabled and curricular differentiation. When we examine how tracking changes in response to educational policies, we see that schools decrease tracking in response to accountability pressures. Finally, when we explore how exposure to tracking correlates with student mobility in the achievement distribution, we find positive effects on high-achieving students with no negative effects on low-achieving students, suggesting that tracking may increase inequality by raising the ceiling.

More →


Jonathan L. Presler.

Using daily lunch transaction data from NYC public schools, I determine which students frequently stand next to one another in the lunch line. I use this `revealed' friendship network to estimate academic peer effects in elementary school classrooms, improving on previous work by defining not only where social connections exist, but the relative strength of these connections. Equally weighting all peers in a reference group assumes that all peers are equally important and may bias estimates by underweighting important peers and overweighting unimportant peers. I find that students who eat together are important influencers of one another's academic performance, with stronger effects in math than in reading. Further exploration of the mechanisms supports my claim that these are friendship networks. I also compare the influence of friends from different periods in the school year and find that connections occurring around standardized testing dates are most influential on test scores.

More →


Monica Lee, James Soland.

Reclassification can be an important juncture in the academic experience of English Learners (ELs). Literature has explored the potential for reclassification to influence academic outcomes like achievement, yet its impact on social-emotional learning (SEL) skills, which are as malleable and important to long-term success, remains unclear. Using a regression discontinuity design, we examine the causal effect of reclassification on SEL skills (self-efficacy, growth mindset, self-management, and social awareness) among 4th to 8th graders. In the districts studied, reclassification improved academic self-efficacy by 0.2 standard deviations for students near the threshold. Results are robust to alternative specifications and analyses. Given this evidence, we discuss ways districts might establish practices that instill more positive academic beliefs among ELs.

More →


Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather C. Hill, Dan Jurafsky.

Responsive teaching is a highly effective strategy that promotes student learning. In math classrooms, teachers might funnel students towards a normative answer or focus students to reflect on their own thinking, deepening their understanding of math concepts. When teachers focus, they treat students’ contributions as resources for collective sensemaking, and thereby significantly improve students’ achievement and confidence in mathematics. We propose the task of computationally detecting funneling and focusing questions in classroom discourse. We do so by creating and releasing an annotated dataset of 2,348 teacher utterances labeled for funneling and focusing questions, or neither. We introduce supervised and unsupervised approaches to differentiating these questions. Our best model, a supervised RoBERTa model fine-tuned on our dataset, has a strong linear correlation of .76 with human expert labels and with positive educational outcomes, including math instruction quality and student achievement, showing the model’s potential for use in automated teacher feedback tools. Our unsupervised measures show significant but weaker correlations with human labels and outcomes, and they highlight interesting linguistic patterns of funneling and focusing questions. The high performance of the supervised measure indicates its promise for supporting teachers in their instruction.

More →


C. Kirabo Jackson, Sebastian Kiguel, Shanette C. Porter, John Q. Easton.

We estimate the longer-run effects of attending an effective high school (one that improves a combination of test scores, survey measures of socio-emotional development, and behaviors in 9th grade) for students who are more versus less educationally advantaged (i.e., likely to attain more years of education based on 8th-grade characteristics). All students benefit from attending effective schools, but the least advantaged students experience larger improvements in high-school graduation, college going, and school-based arrests. This heterogeneity is not solely due to less-advantaged groups being marginal for particular outcomes. Commonly used test-score value-added understates the long-run importance of effective schools, particularly for less-advantaged populations. Patterns suggest this partly reflects less-advantaged students being relatively more responsive to non-test-score dimensions of school quality.

More →