Randomized Controlled Trial (RCT) of ...
Roca, a Program for Justice-Involved Young Men to Prevent Reoffending
Reviewed
Third Sector (August 2024) posted RCT findings for a Massachusetts Pay for Success Project that provided Roca – a violence intervention and behavioral health program – to justice-involved young men to prevent reoffending. This high-quality RCT found no discernible effects on reincarceration or employment over 5 years. The study was able to rule out the sizable positive effects anticipated as the project’s start, but not more modest effects.
The Massachusetts Juvenile Justice Pay for Success (PFS) project is an experiment in a financing model in which the Commonwealth of Massachusetts would pay the costs of a juvenile justice intervention only after it had been shown to have successfully produced the intended benefits . . . .
The intervention financed by the PFS contract was the use of Roca as a reentry intervention for young men released from incarceration. Roca’s four-year intervention consisted of two years of intensive intervention with two years of follow-up. Contracted outcomes of interest, which would generate repayment, were reduced reincarceration and increased employment over the five years following randomization.
The project consisted of a randomized control trial (RCT) in which 1,819 young men were randomized to treatment or control conditions between 2014 and 2018. In addition to estimating the effects of random assignment (RA) to treatment as assigned, the evaluation also estimated the effects of enrollment in the program.
The evaluation design also contemplated the possibility that sample sizes assigned or enrollment might fall short of expectations, thereby reducing statistical power – which indeed occurred. Under these circumstances, the evaluation design called for a supplemental quasi-experimental estimate to be conducted, using a difference-in-differences (DID) design.
The separate RCT and DID estimates of the effects of treatment enrollment were then combined (with the RCT and DID estimates weighted at 80% and 20% respectively) into backstop estimates, which are the final determinants of payment obligations under the Pay for Success contract. The final RCT and DID estimates for employment and incarceration are too small relative to their standard errors to provide confidence in either their direction or magnitude. However, the RCT point-estimates were both in the detrimental direction, while the DID point-estimates were in the beneficial direction. When the combined backstop estimates were calculated, neither the recidivism nor employment backstop outcomes were in the beneficial direction, triggering no payment obligations under the PFS contract.
We have no suggested revisions to the published summary.
No-Spin’s Study Overview
High-quality RCT of Roca, a violence intervention and behavioral health program for justice-involved young men at high risk of reoffending, finds no discernible effects on reincarceration or employment over 5 years. The study was able to rule out the sizable positive effects anticipated at the study’s start but not more modest effects.
Program:
- Roca is a four-year program for justice-involved young men designed to prevent reoffending. It seeks to foster strong relationships with program staff to change destructive thinking patterns, and provides transitional employment and workforce training.
- Under the Pay for Success funding model, private investors (e.g., Goldman Sachs, philanthropic foundations) helped pay for expanded delivery of Roca; and the state agreed to repay them – possibly with a return – only if the program achieved specific, prespecified results as verified through evaluation.
Study Design:
- The study sample comprised 1,819 young men (1,190 treatment vs. 629 control) aged 17 to 24 who were on probation, in the custody of Department of Youth Services, or leaving an adult custodial institution, and were at high risk of reoffending.
- Based on careful review, this was a high-quality RCT (e.g., low sample attrition, prespecified outcomes and analyses).
- However, only 38% of treatment group members actually enrolled in Roca, and 9% of the control group did so, reducing the study’s ability to detect program effects as discussed below.
Findings:
- The study found no discernible impact on reincarceration or employment for the full treatment vs. control group – i.e., “intent-to-treat” impact – over roughly the 5 years after study entry. (The treatment group averaged 13% more days incarcerated than the control group and 2% fewer quarters employed, but these differences weren't statistically significant.)
- The study also estimated Roca’s effects on sample members who actually enrolled in the program, using an analysis that accounts for the rates of non-enrollment in the treatment group and cross-over in the control group noted above. This analysis also found no discernible impacts and was able to rule out a reduction in incarceration for program enrollees over 21%, including the 30-40% reduction anticipated at the project start. However, it couldn't confidently rule out more modest program impacts (i.e., reductions under 21%).
Comment:
- The evaluator also used a prespecified method of combining the RCT results with the results of a non-RCT comparison group analysis, in order to increase the evaluation’s ability to detect possible effects of Roca. This combined analysis, like the RCT analysis, found similar reincarceration and employment outcomes for the treatment versus control group.
- Because the combined analysis didn't show positive impacts, the private investors that helped pay for Roca delivery were not repaid by the state under the Pay for Success agreement.
- Disclosure: Arnold Ventures, which funds No-Spin Evidence Review, helped fund the delivery of Roca as well as the evaluation as part of the Pay for Success project.
Click or tap a highlight to see No-Spin’s comment