Randomized Controlled Trial (RCT) of...

Saga Technology Math Tutoring in High-Poverty High Schools

Reviewed

K-12 Education
Somewhat accurately reported
Credible positive findings

NBER (May 2024) posted an RCT of Saga Technology, a program that provides 9th graders with high-dosage math tutoring and engagement with educational technology. This large, high-quality RCT found sizable gains (~86% of a grade-level) in math achievement on end-of-year district tests.

High-dosage tutoring is an effective way to improve student learning (Nickow et al., 2024; Guryan et al., 2023). Finding ways to deliver high-dosage tutoring at large scale remains a challenge. Two primary challenges to scaling are cost and staffing. One possible solution is to reduce costs by substituting some tutor time with computer-assisted learning (CAL) technology. The question is: Does doing so compromise effectiveness? This paper provides evidence from a randomized controlled trial (RCT) of approximately 4,000 students in two large school districts in 2018- 19 and 2019-20. The RCT tested the effectiveness of an in-school math tutoring program where students worked in groups of four, with two students working with an in-person tutor while the other two worked on CAL, alternating every other day. The tutoring model had per-pupil costs approximately 30 percent lower than the 2- to-1 tutoring model studied in Guryan et al. (2023).  We find gains in students’ math standardized test scores of 0.23 standard deviations for participating students,   which are almost as large as the effect sizes of the 2-to-1 tutoring model reported in Guryan et al. (2023). These findings suggest strategic use of technology may be a way to increase the scalability of HDT.

Full published study

High-dosage tutoring is an effective way to improve student learning (Nickow et al., 2024; Guryan et al., 2023). Finding ways to deliver high-dosage tutoring at large scale remains a challenge. Two primary challenges to scaling are cost and staffing. One possible solution is to reduce costs by substituting some tutor time with computer-assisted learning (CAL) technology. The question is: Does doing so compromise effectiveness? This paper provides evidence from a randomized controlled trial (RCT) of approximately 4,000 students in two large school districts in 2018- 19 and 2019-20. The RCT tested the effectiveness of an in-school math tutoring program where students worked in groups of four, with two students working with an in-person tutor while the other two worked on CAL, alternating every other day. The tutoring model had per-pupil costs approximately 30 percent lower than the 2- to-1 tutoring model studied in Guryan et al. (2023). We find gains in students’ math standardized test scores of 0.23 standard deviations for participating students,  that the offer of program participation in ninth grade produced gains in students’ math standardized test scores of 0.19 standard deviations at the end of the school year (intent-to-treat effect). For those students who took up the offer, we find gains of 0.23 standard deviations (treatment-on-treated effect),   which are almost as large as the effect sizes of the 2-to-1 tutoring model reported in Guryan et al. (2023). These findings suggest strategic use of technology may be a way to increase the scalability of HDT. 

No-Spin’s Study Overview

High-quality RCT of the Saga Technology high-dosage math tutoring program for 9th graders in New York City and Chicago finds sizable gain (~86% of a grade-level) in math achievement on end-of-year district tests.

Program:

  • Students in the program alternate (on successive days) between working with an in-person, trained tutor and individually engaging with an education technology platform. The student-tutor ratio is 4:1. The program takes place 50 minutes a day, 5 days a week, for the full 9th grade school year.

Study Design:

  • The study sample comprised 2,065 incoming 9th graders in 6 high-poverty high schools (3 in NYC, 3 in Chicago). 81% were Black or Hispanic; 91% were low-income. Based on careful review, this was a well-conducted RCT (e.g., baseline balance, low attrition, preregistered outcomes and analyses).

Findings:

  • The study found a sizable, statistically-significant impact on end-of-year math scores on district tests – the study’s primary outcome. The effect size was 0.19 - which represents approximately an 86% improvement over the annual gain in math otherwise expected for 9th graders.
  • The study also found significantly improved math GPA and fewer math course failures through fall of 10th grade, after which COVID interrupted study follow-up.

Comment:

  • Prior RCTs have found sizable, sustained effects for Saga's traditional program (2:1 tutoring ratio, no tech component). Saga Technology's impacts are almost as big, but the cost is 40% lower (around $2600 per tutored student).
  • Taken together, these studies constitute strong, replicated RCT evidence - providing confidence that schools which adopt and faithfully implement Saga's traditional or technology model will see meaningful math gains for low-income students.
  • Disclosure: Arnold Ventures, which funds the No-Spin Evidence Review, helped fund the Saga Technology study.

Click or tap a highlight to see No-Spin’s comment

Related Reviews