Randomized Controlled Trial (RCT) of...
Khoaching with Khan Academy to Support Grade 3-8 Math Learning
Reviewed
NBER (April 2024) posted an RCT of Khoaching with Khan Academy – a program integrating computer-assisted learning into classroom math teaching. Despite the positive portrayal of results in the study’s abstract, it found no discernible impact on the prespecified primary outcome – grades 3-8 math scores on the state test after one year.
Mastery learning - the process by which students must demonstrated proficiency with a single topic before moving on - is well recognized as one of the most effective ways to learn, yet many teachers struggle or remain unsure about how to implement it into a classroom setting. This study evaluates a program to encourage greater mastery learning through technology and proactive continuous teacher support. Focusing on elementary and middle school mathematics, teachers receive weekly coaching in how to use Computer Assisted Learning (CAL) for helping students follow a customized roadmap of incremental progress. Results from two field experiments indicate significant Intent To Treat effects on math performance of 0.12 - 0.22 standard deviations. Further analysis indicates that these gains are from students in classrooms with at least an average of 35 minutes of practice per week. Teachers able to achieve high-dosage practice have a high degree of initial buy-in, a clear implementation strategy for when practice occurs, and a willingness to closely monitor progress and follow-up with struggling students.
Mastery learning - the process by which students must demonstrate proficiency with a single topic before moving on - is well recognized as one of the most effective ways to learn, yet many teachers struggle or remain unsure about how to implement it into a classroom setting. This study evaluates a program to encourage greater mastery learning through technology and proactive continuous teacher support. Focusing on elementary and middle school mathematics, teachers receive weekly coaching in how to use Computer Assisted Learning (CAL) for helping students follow a customized roadmap of incremental progress. Results from two field experiments indicate significant Intent To Treat effects on math performance of 0.12 - 0.22 standard deviations.
A randomized controlled trial in the Arlington Independent School District with students in grades 3-8 found no discernible effects on the study’s pre-registered primary outcome – students’ end-of-year math scores on the state test (the effect, 0.025 standard deviations, was not statistically significant.) Exploratory analyses suggest possible gains
Further analysis indicates that these gains are from for students in classrooms with at least an average of 35 minutes of practice per week. Teachers able to achieve high-dosage practice have a high degree of initial buy-in, a clear implementation strategy for when practice occurs, and a willingness to closely monitor progress and follow-up with struggling students.
No-Spin’s Study Overview
High-quality RCT of Khoaching with Khan Academy (KWIK) finds no discernible impact on grade 3-8 math scores on the state test after one year.
Program:
- KWiK combines Khan Academy — a computer assisted learning platform in which students watch short videos, complete short exercises, get feedback, etc. — with weekly coaching for teachers on using the platform as part of their curriculum.
Study Design:
- The study randomly assigned 224 interested math teachers in grades 3-8 in Arlington, TX schools (grouped together with other teachers in same school and grade) to treatment vs control. The large majority of treatment-group teachers attended the virtual training sessions and met with their coach.
- Based on careful review, this was a high-quality RCT (e.g., baseline balance, low attrition, valid analyses).
Findings:
- The study found no discernible impact on the primary prespecified outcome: students’ end-of-year scores on the state math test. (The non-significant effect size was .025.)
- The study reports positive math impacts in grades 3-6, as opposed to grades 7-8. But this was a post-hoc (not prespecified) subgroup analysis that isn’t reliable under established scientific standards (IES, FDA) as it can easily yield false-positive/chance findings.
- The study and a related quasi-experiment found suggestive evidence that positive impacts occurred when teachers gave students sufficient Khan Academy practice time (e.g., at least 35 minutes a week). This seems like a good hypothesis to test in future RCTs.
Click or tap a highlight to see No-Spin’s comment