Randomized Controlled Trial (RCT) of...

Choose to Change, a Group Therapy and Mentoring Program for At-risk Teens

Reviewed

Crime / Justice
Not accurately reported

The Annenberg Institute (October 2024) posted an RCT of Choose to Change, a group therapy and mentoring program for at-risk teens. Despite the study abstract’s claimed effects on youth crime, the study found no discernible impact on its preregistered primary measure of crime – number of youth arrests – over 3 years.

Racial disparities in violence exposure and criminal justice contact are a subject of growing policy and public concern. We conduct a  large-scale, randomized controlled trial  of a six-month behavioral health intervention combining intensive mentoring and group therapy designed to reduce criminal justice and violence involvement among Black and Latinx youth in Chicago.  Over 24 months, youth offered the program experienced an 18 percent reduction in the probability of any arrest and a 23 percent reduction in the probability of a violent-crime arrest. These statistically significant impacts, with smaller magnitudes, continue to persist up to 3 years post randomization.   To better understand the behavior change we observe given an arrest is a proxy for criminal behavior, we create a supervised machine learning algorithm from arrest narratives that determines if an arrest was initiated more or less at the discretion of police. We find that the program’s impacts are concentrated in arrests where officers have less discretion in initiating contact, while having little impact on more discretionary contact arrests (e.g. a young person exhibiting “suspicious” behavior). This analysis suggests the effects of the program are being driven by a reduction in youth offending behavior rather than by avoiding police contact.

Full published study

Racial disparities in violence exposure and criminal justice contact are a subject of growing policy and public concern. We conduct a large-scale, randomized controlled trial of a six-month behavioral health intervention combining intensive mentoring and group therapy designed to reduce criminal justice and violence involvement among Black and Latinx youth in Chicago. Over 24 months, youth offered the program experienced an 18 percent reduction in the probability of any arrest and a 23 percent reduction in the probability of a violent-crime arrest. These statistically significant impacts, with smaller magnitudes, continue to persist up to 3 years post randomization.  We found no statistically significant impact on our preregistered primary measure of criminal activity – number of youth arrests – over 3 years. We found a significant effect on an exploratory outcome - percent of youth arrested over 3 years (33.4% of Choose to Change youth compared to 38.6% of control youth) – that may warrant examination in future research.  To better understand the behavior change we observe given an arrest is a proxy for  this exploratory effect on criminal behavior , we create a supervised machine learning algorithm from arrest narratives that determines if an arrest was initiated more or less at the discretion of police. We find that the program’s impacts are concentrated in arrests where officers have less discretion in initiating contact, while having little impact on more discretionary contact arrests (e.g. a young person exhibiting “suspicious” behavior). This analysis suggests the effects of the program are being driven by a reduction in youth offending behavior rather than by avoiding police contact.

No-Spin’s Study Overview

High-quality RCT of Choose to Change, a group therapy and mentoring program for at-risk teens, finds no discernible impact on the number of youth arrests over 3 years – the study’s preregistered primary measure of criminal activity. (It found a suggestive effect on the probability of arrest that may warrant examination in future studies.)

Program and Study Design:

  • Choose to Change (C2C) is a 6-month behavioral health program of intensive mentoring and group therapy for at-risk Chicago youth age 13-18.
  • The study randomized 2,074 youth to C2C vs control, and was well-conducted based on our careful review (e.g., low attrition, baseline balance).

Findings:

  • The study found no discernible impact on the primary preregistered outcome - average number of youth arrests - measured over 3 years. (C2C group averaged 1.091 arrests vs 1.175 for controls; the difference wasn't statistically significant).
  • On secondary outcomes: 1 of 16 impacts was statistically significant.
  • On other outcomes: The study found a significant effect on the percent of youth arrested over 3 years (33.4% C2C vs 38.6% control). But this was a post-hoc outcome - not prespecified as primary or secondary - and one of multiple post-hoc outcomes that the study measured (along with crime victimizations, police stops, etc.).
  • Thus, the finding on percent arrested is only suggestive. It could be a true effect, but isn't reliable under established standards (FDA, IES) until confirmed in future studies. Such confirmation is needed to rule out the possibility that it's a chance finding resulting from the study's measurement of many outcomes.

Comment:

  • The study preregistered two other primary outcomes – school engagement and employment – that it will report on in the future, once the outcome data are available.
  • Arnold Ventures, which funds the No-Spin Evidence Review, helped fund the Choose to Change RCT.

Click or tap a highlight to see No-Spin’s comment

Related Reviews