Relevant Excerpt from the Outcomes Section:
Reported PFS [Pay for Success] Findings
Exhibit 3 provides the estimated ITT and IV Population Outcomes (see Appendix N for details). As shown, the Phase I results do not provide evidence of a positive effect of CEO services on the targeted PFS outcomes. As the Project’s Independent Validator, I certify that these calculations were conducted in accordance with the agreed upon methodology as described in Schedule I of the PFS Agreement.
In terms of recidivism, on average there was only a 3-day difference in the rates of incarceration while under Community Supervision, and a two percentage point difference in the estimated fraction with remaining sentences. In both cases the results were “worse” for the Treatment Group with participants serving a larger number of bed days over the period of Phase I. The IV estimates – adjusting for observed rates of CEO service enrollment – are, as expected, slightly higher but not qualitatively different from the ITT estimates.
With regard to employment, the Phase I results show a less than one percentage point difference in the rate of employment in the 4th quarter after parole. The IV estimate are again larger with an estimated employment difference of two percentage points.
We have no suggested revisions to the study’s Outcomes Section.
No-Spin’s Study Overview
High-quality RCT of Center for Employment Opportunities (CEO)’s program of subsidized transitional jobs and other employment supports for high-risk male parolees finds no effects on reincarceration or employment outcomes. Although the estimated effects are near zero, the study sample may not have been large enough to rule out modest effects.
Program:
- CEO provided high-risk adult male parolees with a four-to-six month program that included life skills training and placement in a subsidized transitional job, followed by placement assistance for unsubsidized jobs, post-placement counseling, and other work supports.
- Under a Pay for Success funding model, private investors helped pay for expanded delivery of CEO; and government agreed to repay them – possibly with a return – only if the program achieved specific, prespecified results as verified through evaluation.
Study Design:
- The study sample comprised 2,357 adult men who were about to be released from prison to community supervision in New York City or Rochester, New York, and had been identified as high risk for reincarceration based on an assessment tool.
- The men were randomly assigned to a treatment group (N=1,502) that was referred to CEO services following release, or a control group (N=855) that was not.
- 71% of the treatment group received at least some CEO services, as did 29% of the control group. This imperfect adherence to the assigned conditions somewhat reduced the study’s ability to detect program effects. However, the resulting treatment-control contrast of 42 percentage points exceeded the minimum 30 point contrast that stakeholders had prespecified as necessary for the RCT to “work” – i.e., to have a reasonable chance of affecting the targeted outcomes.
- Based on our review criteria, this was a high-quality RCT.
Findings on the Two Primary Outcomes:
- The study found no effect on reincarceration, measured as the number of days spent in, or sentenced to, jail or prison (“bed days”) over an average follow-up of 22 months post-prison release (average bed days were 233 in the treatment group versus 230 in the control group).
- The study found no effect on employment in the fourth quarter post-release (17.6% in the treatment group versus 16.8% in the control group were employed).
- The study sample may not have been large enough to rule out modest program effects. However, this can’t be directly assessed because, per agreement among the project stakeholders, the study did not report confidence intervals for the estimated impacts.
Comment:
- Although the study report was finalized in 2017 and briefly summarized and cited in a 2020 publication of the U.S. Department of Labor (DOL), we only recently obtained the full unpublished report from the study author, enabling us to review it for No-Spin.
- Because the study didn't find positive impacts, the private investors that helped pay for CEO delivery were not repaid by DOL under the Pay for Success agreement.
Disclosure: Arnold Ventures, which funds No-Spin Evidence Review, was one of the private investors in the CEO Pay for Success initiative.
1 Puma, M. (March 2017). New York State Pay for Success (PFS) Project: "Employment to Break the Cycle of Re-Incarceration" Phase I Validation Report. Mike Puma Associates LLC.
Click or tap a highlight to see No-Spin’s comment