Evidence Reports

CAPR (October 2023) published results of a New York RCT of Multiple Measures Assessment (MMA) – a data analytic tool used by community colleges to place incoming students into college-level versus developmental (i.e., remedial) courses. Despite the mostly positive portrayal of results in the study’s abstract, it found negligible impacts on the preregistered primary measures of student success at the 4.5-year mark.

Annenberg Institute (January 2025) posted a comparison-group study of a California program to train teachers in the “science of reading.” Despite the study abstract’s claims of positive effects, the study found no discernible impact on 3rd grade English Language Arts (ELA) achievement in its primary analysis.

Quarterly Journal of Economics (Nov. 2024) published an RCT of Sit-D, a police training program teaching the value of developing multiple perspectives on a situation. Despite the study’s claimed effects on officer performance in the field, it doesn’t report findings for most of its preregistered primary outcomes, preventing any reliable conclusions about program effectiveness.

AERA Open published long-term RCT findings for Early Colleges in North Carolina (providing grade 9-13 students an opportunity to earn an associate degree or college credit during high school). Despite claims of large effects in the study’s abstract, its primary finding was a modest effect (2.4 percentage points, near statistically significant) on the rate of bachelor’s degree completion at the 10-year mark.

The Annenberg Institute (October 2024) posted an RCT of Choose to Change, a group therapy and mentoring program for at-risk teens. Despite the study abstract’s claimed effects on youth crime, the study found no discernible impact on its preregistered primary measure of crime – number of youth arrests – over 3 years.

An RCT of a cash transfer program for individuals experiencing homelessness, published in the August 2023 Proceedings of the National Academies of Sciences (PNAS), portrays its results as positive despite finding no discernible impact on any of the study’s pre-registered primary or secondary outcomes.

NBER (April 2024) posted an RCT of Khoaching with Khan Academy – a program integrating computer-assisted learning into classroom math teaching. Despite the positive portrayal of results in the study’s abstract, it found no discernible impact on the prespecified primary outcome – grade 3-8 math scores on the state test after one year.

An RCT of the Chicago Heights (CHECC) preschool program, published in Journal of Political Economy (December 2024), portrays results of its grade-school follow-up as positive despite finding no impact on the preregistered primary outcome – state test scores in English and math.

Development & Psychopathology (December 2022) published an RCT of a program that provides home visiting and doula visits to low-income women starting in pregnancy. Despite the positive portrayal of results in the study’s abstract, the study found no discernible impact on child outcomes and little impact on parent practices, at age 30 months.