Evidence Reports

New England Journal of Medicine (June 2024) published an RCT of Communities that Heal, a data-driven program that engaged communities in four states to rapidly deploy evidence-based practices to reduce opioid overdose deaths. This large, high-quality RCT found no discernible impact on opioid overdose deaths over 1 year.

Annenberg Institute (January 2025) posted a comparison-group study of a California program to train teachers in the “science of reading.” Despite the study abstract’s claims of positive effects, the study found no discernible impact on 3rd grade English Language Arts (ELA) achievement in its primary analysis.

Quarterly Journal of Economics (Nov. 2024) published an RCT of Sit-D, a police training program teaching the value of developing multiple perspectives on a situation. Despite the study’s claimed effects on officer performance in the field, it doesn’t report findings for most of its preregistered primary outcomes, preventing any reliable conclusions about program effectiveness.

AERA Open published long-term RCT findings for Early Colleges in North Carolina (providing grade 9-13 students an opportunity to earn an associate degree or college credit during high school). Despite claims of large effects in the study’s abstract, its primary finding was a modest effect (2.4 percentage points, near statistically significant) on the rate of bachelor’s degree completion at the 10-year mark.

Third Sector posted (August 2024) RCT findings for a Massachusetts Pay for Success Project that provided Roca – a violence intervention and behavioral health program – to justice-involved young men to prevent reoffending. This high-quality RCT found no discernible effects on reincarceration or employment over 5 years. The study was able to rule out the sizable positive effects anticipated as the project’s start, but not more modest effects.

Mobility posted (October 2024) long-term RCT results for Project QUEST's occupational training of low-income adults for well-paying healthcare jobs. This high-quality RCT found a 15-20% gain in annual earnings at the 14 year follow-up - an extremely promising (albeit not quite definitive) result.

The Annenberg Institute (October 2024) posted an RCT of Choose to Change, a group therapy and mentoring program for at-risk teens. Despite the study abstract’s claimed effects on youth crime, the study found no discernible impact on its preregistered primary measure of crime – number of youth arrests – over 3 years.

An RCT of a cash transfer program for individuals experiencing homelessness, published in the August 2023 Proceedings of the National Academies of Sciences (PNAS), portrays its results as positive despite finding no discernible impact on any of the study’s pre-registered primary or secondary outcomes.

NBER (April 2024) posted an RCT of Khoaching with Khan Academy – a program integrating computer-assisted learning into classroom math teaching. Despite the positive portrayal of results in the study’s abstract, it found no discernible impact on the prespecified primary outcome – grade 3-8 math scores on the state test after one year.