Search EdWorkingPapers

Search for EdWorkingPapers here by author, title, or keywords.

Methodology, measurement and data

Matthew D. Baird, John Engberg, Isaac M. Opper.

We consider the case in which the number of seats in a program is limited, such as a job training program or a supplemental tutoring program, and explore the implications that peer effects have for which individuals should be assigned to the limited seats. In the frequently-studied case in which all applicants are assigned to a group, the average outcome is not changed by shuffling the group assignments if the peer effect is linear in the average composition of peers. However, when there are fewer seats than applicants, the presence of linear-in-means peer effects can dramatically influence the optimal choice of who gets to participate. We illustrate how peer effects impact optimal seat assignment, both under a general social welfare function and under two commonly used social welfare functions. We next use data from a recent job training RCT to provide the first evidence of large peer effects in the context of job training for disadvantaged adults. Finally, we combine the two results to show that the program's effectiveness varies greatly depending on whether the assignment choices account for or ignore peer effects.

More →


Peter Q. Blair, Papia Debroy, Justin Heck.

Over the past four decades, income inequality grew significantly between workers with bachelor’s degrees and those with high school diplomas (often called “unskilled”). Rather than being unskilled, we argue that these workers are STARs because they are skilled through alternative routes—namely their work experience. Using the skill requirements of a worker’s current job as a proxy of their actual skill, we find that though both groups of workers make transitions to occupations requiring similar skills to their previous occupations, workers with bachelor’s degrees have dramatically better access to higher wage occupations where the skill requirements exceed the workers’ observed skill. This measured opportunity gap offers a fresh explanation of income inequality by degree status and reestablishes the important role of on-the-job-training in human capital formation.

More →


Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather C. Hill, Dan Jurafsky, Tatsunori Hashimoto.

In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers' uptake of student contributions has been linked to higher student achievement. Yet measuring and improving teachers' uptake at scale is challenging, as existing methods require expensive annotation by experts. We propose a framework for computationally measuring uptake, by (1) releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts; (2) formalizing uptake as pointwise Jensen-Shannon Divergence (pJSD), estimated via next utterance classification; (3) conducting a linguistically-motivated comparison of different unsupervised measures and (4) correlating these measures with educational outcomes. We find that although repetition captures a significant part of uptake, pJSD outperforms repetition-based baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation. We apply our uptake measure to three different educational datasets with outcome indicators. Unlike baseline measures, pJSD correlates significantly with instruction quality in all three, providing evidence for its generalizability and for its potential to serve as an automated professional development tool for teachers.

More →


Daniel Kreisman, Jonathan Smith, Bondi Arifin.

How do college non-completers list schooling on their resumes? The negative signal of not completing might outweigh the positive signal of attending but not persisting. If so, job-seekers might hide non-completed schooling on their resumes. To test this we match resumes from an online jobs board to administrative educational records. We find that fully one in three job-seekers who attended college but did not earn a degree omit their only post-secondary schooling from their resumes. We further show that these are not casual omissions but are strategic decisions systematically related to schooling characteristics, such as selectivity and years of enrollment. We also find evidence of lying, and show which degrees listed on resumes are most likely untrue. Lastly, we discuss implications. We show not only that this implies a commonly held assumption, that employers perfectly observe schooling, does not hold, but also that we can learn about which college experiences students believe are most valued by employers.

More →


Sophie Litschwartz, Luke Miratrix.

In multisite experiments, we can quantify treatment effect variation with the cross-site treatment effect variance. However, there is no standard method for estimating cross-site treatment effect variance in multisite regression discontinuity designs (RDD). This research rectifies this gap in the literature by systematically exploring and evaluating methods for estimating the cross-site treatment effect variance in multisite RDDs. Specifically, we formalize a fixed intercepts/random coefficients (FIRC) RDD model and develop a random effects meta-analysis (Meta) RDD model for estimating cross-site treatment effect variance. We find that a restricted FIRC model works best when the running variables' relationship to the outcome is stable across sites but can be biased otherwise. In those instances, we recommend using either the unrestricted FIRC model or the meta-analysis model; with the unrestricted FIRC model generally performing better when the average number of in-bandwidth observations is less than 120 and the meta-analysis model performing better when the average number of in-bandwidth observations is above 120. We apply our models to a high school exit exam policy in Massachusetts that required students who passed the high school exit exam but were still determined to be nonproficient to complete an ``Education Proficiency Plan" (EPP). We find the EPP policy had a positive local average treatment effect on whether students completed a math course their senior year on average across sites, but that the impact varied enough such that a third of schools could have had a negative impact.

More →


Susana Claro, Valentina Paredes, Verónica Cabezas, Gabriel Cruz.

Growing evidence shows that a student's growth mindset (the belief that intelligence is malleable) can benefit their academic achievement. However, due to limited information, little is known about how a teachers’ growth mindset affects their students’ academic achievement. In this paper, we study the impact of teacher growth mindset on academic achievement for a nationwide sample of 8th and 10th grade students in Chile in 2017. Using a student fixed effect model that exploits data from two subject teachers for each student, we find that being assigned to a teacher with a growth mindset increases standardized test scores by approximately 0.02 standard deviations, with larger effects on students with high GPAs and particularly on students in low socioeconomic schools.

More →


Brendan Bartanen, Aliza N. Husain.

A growing literature uses value-added (VA) models to quantify principals' contributions to improving student outcomes. Principal VA is typically estimated using a connected networks model that includes both principal and school fixed effects (FE) to isolate principal effectiveness from fixed school factors that principals cannot control. While conceptually appealing, high-dimensional FE regression models require sufficient variation to produce accurate VA estimates. Using simulation methods applied to administrative data from Tennessee and New York City, we show that limited mobility of principals among schools yields connected networks that are extremely sparse, where VA estimates are either highly localized or statistically unreliable. Employing a random effects shrinkage estimator, however, can alleviate estimation error to increase the reliability of principal VA.

More →


Lina Anaya, Nagore Iriberri, Pedro Rey-Biel, Gema Zamarro.

Standardized assessments are widely used to determine access to educational resources with important consequences for later economic outcomes in life. However, many design features of the tests themselves may lead to psychological reactions influencing performance. In particular, the level of difficulty of the earlier questions in a test may affect performance in later questions. How should we order test questions according to their level of difficulty such that test performance offers an accurate assessment of the test taker's aptitudes and knowledge? We conduct a field experiment with about 19,000 participants in collaboration with an online teaching platform where we randomly assign participants to different orders of difficulty and we find that ordering the questions from easiest to most difficult yields the lowest probability to abandon the test, as well as the highest number of correct answers. Consistent results are found exploiting the random variation of difficulty across test booklets in the Programme for International Student Assessment (PISA), a triannual international test, for the years of 2009, 2012, and 2015, providing additional external validity. We conclude that the order of the difficulty of the questions in tests should be considered carefully, in particular when comparing performance between test-takers who have faced different order of questions.

More →


Jeffrey T. Denning, Eric R. Eide, Kevin J. Mumford, Daniel Sabey.

After increasing in the 1970s and 1980s, time to bachelor’s degree has declined since the 1990s. We document this fact using data from three nationally representative surveys. We show that this pattern is occurring across school types and for all student types. Using administrative student records from 11 large universities, we confirm the finding and show that it is robust to alternative sample definitions. We discuss what might explain the decline in time to bachelor’s degree by considering trends in student preparation, state funding, student enrollment, study time, and student employment during college.

More →


M. Danish Shakeel, Paul E. Peterson.

Principals (policy makers) have debated the progress in U. S. student performance for a half century or more. Informing these conversations, survey agents have administered seven million psychometrically linked tests in math and reading in 160 waves to national probability samples of selected cohorts born between 1954 and 2007. This study is the first to assess consistency of results by agency. We find results vary by agent, but consistent with Flynn effects, gains are larger in math than reading, except for the most recent period. Non-whites progress at a faster pace. Socio-economically disadvantaged white, black, and Hispanic students make greater progress when tested in elementary school, but that advantage attenuates and reverses itself as students age. We discuss potential moderators.

More →