A Non-RCT “Digital Twins” Method to Evaluate Job Training Programs, Proposed in Lieu of RCTs

Reviewed

Workforce Development
Somewhat accurately reported

Chetty et. al. (November 2026) posted a study examining whether a new non-RCT “Digital Twins” method to evaluate US job training programs can match the results of 37 previously-conducted RCTs. While the study found impressive correlation between Digital Twins and RCT results, additional confirmatory research – in which all method parameters and procedures for testing it against additional RCTs are publicly prespecified – is needed to establish whether it can reliably reproduce RCT results and is ready for practical use.

Modern program evaluation prioritizes evidence from randomized controlled trials (RCTs) because prior research found that observational estimates of program impacts often diverge substantially from experimental benchmarks. We revisit this finding — which was based on small survey datasets — in the era of large-scale administrative data.  Using anonymized income tax records covering all American adults between 1999 and 2025, we estimate the effects of workforce training programs on earnings by using machine learning models to predict counterfactuals from rich data on prior work histories, demographics, and residential locations. Rather than testing whether our observational estimates are unbiased, we use 37 RCTs to estimate the distribution of their bias. We find that our machine learning estimates closely match the RCT estimates, with root mean squared error (RMSE) comparable to that of the RCTs themselves.  Our method reduces RMSE by 60% relative to parametric estimators used in prior work because our larger samples and richer covariates permit more flexible, higher-dimensional models of the counterfactual. Applying our method to a broader set of sites and enrollment years beyond the experimental sample, we find that treatment effects on earnings vary substantially across locations and over time, even within a given program. Programs that place more participants into target-sector jobs generate the largest earnings gains, with impacts that persist even after participants leave their initial jobs. These findings provide guidance for improving job training programs and, more broadly, illustrate a scalable approach to program evaluation that uses observational data to construct  experimentally validated  “digital twins.”

Full Study Report

Modern program evaluation prioritizes evidence from randomized controlled trials (RCTs) because prior research found that observational estimates of program impacts often diverge substantially from experimental benchmarks. We revisit this finding — which was based on small survey datasets — in the era of large-scale administrative data. Using anonymized income tax records covering all American adults between 1999 and 2025, we estimate the effects of workforce training programs on earnings by using machine learning models to predict counterfactuals from rich data on prior work histories, demographics, and residential locations. Rather than testing whether our observational estimates are unbiased, we use 37 RCTs to estimate the distribution of their bias. We find that our machine learning estimates closely match the RCT estimates, with root mean squared error (RMSE) comparable to that of the RCTs themselves. Our method reduces RMSE by 60% relative to parametric estimators used in prior work because our larger samples and richer covariates permit more flexible, higher-dimensional models of the counterfactual. Applying our method to a broader set of sites and enrollment years beyond the experimental sample, we find that treatment effects on earnings vary substantially across locations and over time, even within a given program. Programs that place more participants into target-sector jobs generate the largest earnings gains, with impacts that persist even after participants leave their initial jobs.  An important next step to validate our method will be to test it in confirmatory research in which all method parameters and procedures for testing it against additional RCTs are publicly prespecified. If validated, these findings would provide guidance for improving job training programs and, more broadly, illustrate a scalable approach to program evaluation that uses observational data to construct experimentally validated “digital twins.”

No-Spin’s Study Overview

A study by Chetty et. al. finds that a non-RCT “Digital Twins” method for evaluating US job training programs produces results that largely mirror those of 37 previously conducted RCTs. While the correlation is impressive, additional confirmatory research – in which all method parameters and procedures for testing it against additional RCTs are publicly prespecified – is needed to establish whether the method can reliably reproduce RCT results and is ready for practical use in lieu of RCTs.

The “Digital Twin” Method:

  • Per the authors’ study summary: “Digital Twins is a new method of estimating the impacts of workforce training and other career advancement programs…. The method applies machine learning to anonymized Census and tax records covering all American adults to identify ‘digital twins’ for program participants: people with similar backgrounds and work histories who did not enroll.”
  • Digital Twins’ potential advantage over other non-RCT methods – which have often failed to reproduce RCT results [1, 2] – is that it “draws on far larger samples, much richer data, and modern machine learning.”
  • If validated, Digital Twins would offer a new way to evaluate job training programs that is generally less expensive and faster than running RCTs and could therefore be an important new tool for evidence-based policymaking.
  • Per the authors, future RCTs would be used to periodically validate and recalibrate the Digital Twins method, rather than as a primary means of evaluating job training programs.  

Study Design and Findings:

  • The researchers compared results from the Digital Twins method with those of 37 RCTs of US workforce training programs conducted between 2000 and 2019.
  • They found a high correlation (0.93) between the Digital Twins and RCT estimated impacts on participant earnings, as well as similar overall estimation error.

What More Is Needed:

  • To establish whether the Digital Twins method can reliably reproduce RCT results, we believe additional confirmatory research is needed.
  • The purpose is to address a central challenge in studies comparing non-RCT methods to RCTs, described in Bloom et. al., MDRC 2002: “Researchers know the right answer in advance” – that is, they know the RCT results they’re trying to replicate – and “it is possible to continue testing [non-RCT] estimators until one finds something that works. Unfortunately, doing so runs the risk of basing findings on chance relationships in the data for a specific sample.”
  • Importantly, researchers may do this inadvertently, producing a strong correlation between the non-RCT and RCT results that is a statistical mirage which would not hold up in testing against a new set of RCTs.
  • To address this challenge, Bloom et. al. – in their study comparing non-RCTs to RCTs in welfare-to-work programs – fully prespecified their non-RCT methods, as well as the procedures for testing them, under the guidance of an expert advisory panel. Separately, Cook, Shadish, and Wong (2008) recommended blinding analysts of non-RCT and RCT data to each other’s results.  

Recommended Next Steps:

  • For the Digital Twins method, we believe it is important to address the above challenge in a definitive, independently verifiable manner before the method is used in lieu of RCTs to evaluate workforce training programs. We recommend that researchers take the following next steps under guidance from an expert panel:
    • Fully prespecify the following items in a manner that doesn’t leave room for post-hoc researcher discretion: (i) the Digital Twins estimator (including items such as the choice of covariates, model specifications, sample restrictions, and hyperparameter-tuning procedures); and (ii) procedures for testing the estimator against a new set of RCTs (including the list of RCTs and the impact estimates within each RCT that they seek to reproduce). Ideally, this new set of RCTs would include a few studies that are nearing completion but haven’t yet reported findings, so that the researchers are blind to the results.
    • Publicly register their prespecified approach prior to initiating the above tests against the new RCTs, consistent with established open science standards (e.g., Institute of Education Sciences, International Committee of Medical Journal Editors).
    • After testing occurs, publicly report findings for all prespecified comparisons between the Digital Twins and the RCT results.

Click or tap a highlight to see No-Spin’s comment