Randomized Controlled Trial (RCT) of...
Multiple Measures Assessment (MMA) to Improve Student Outcomes at New York Community Colleges
Reviewed
CAPR (October 2023) published results of a New York RCT of Multiple Measures Assessment (MMA) – a data analytic tool used by community colleges to place incoming students into college-level versus developmental (i.e., remedial) courses. Despite the mostly positive portrayal of results in the study’s abstract, it found negligible impacts on the preregistered primary measures of student success at the 4.5-year mark.
We find that the MMA method used at the colleges improved access to and success in college-level courses and that lower cut scores in English rather than math are associated with larger and longer lasting impacts on completion of college-level coursework. While MMA improved outcomes among student subgroups, it had little to no impact on gaps in outcomes between subgroups. We also find that bumped-up students had substantially better outcomes in both math and English, while bumped-down students had substantially worse outcomes. Our results suggest that increased access to college-level courses is the driving factor in the positive outcomes experienced by program group students and that placement into standalone developmental courses can have detrimental effects on student outcomes. In the discussion of the study’s results, we make recommendations for adopting MMA at colleges. Implemented together with other initiatives to support students, MMA can be a first step on the path to success for incoming students.
We find that the MMA method used at the colleges
had little impact on student success after nine semesters, including the rate of credential completion or transfer to a four-year institution, based on our primary analysis. Our exploratory analyses produced the following additional findings that would be useful to test in future research. For the subgroup of students that MMA bumped up into college-level coursework, we found that MMA
improved access to and success in college-level courses and that lower cut scores in English rather than math are associated with larger and longer lasting impacts on completion of college-level coursework. While MMA improved outcomes among student subgroups, it had little to no impact on gaps in outcomes between subgroups. We also find that bumped-up students had substantially better outcomes in both math and English, while bumped-down students had substantially worse outcomes. Our results suggest that increased access to college-level courses is the driving factor in the positive outcomes experienced by program group students and that placement into standalone developmental courses can have detrimental effects on student outcomes. In the discussion of the study’s results, we make recommendations for adopting MMA at colleges. Implemented together with other initiatives to support students, MMA
may
can be a first step on the path to success for incoming students.
No-Spin’s Study Overview
Large, high-quality RCT of MMA as implemented at seven New York community colleges finds little impact on the study’s preregistered primary measures of student success, including the rate of credential completion or transfer to a four-year institution, 4.5 years after random assignment.
Program:
- MMA as implemented by the colleges in this study used predictive algorithms – based on high school GPA, placement test scores, time since high school graduation, and other factors – to place students into developmental (i.e., remedial) versus college-level courses. Colleges' traditional procedure is to use only a standardized test to determine placement.
Study Design:
- The study randomly assigned 12,796 incoming students at 7 State University of New York community colleges to placement using MMA (treatment) versus standardized test (control). Use of MMA changed the placement of 26% of students in math and 51% of students in English, compared to their control group counterparts.
- Based on our careful review, this was a well-conducted RCT (e.g., baseline balance, negligible attrition, successful program implementation, pre-registered analyses).
Findings:
- Despite changing student placements, MMA had little impact on the primary study outcomes at the 4.5 year follow-up: percent of students completing college-level math (40.5% treatment vs 39.3% control) and English (52.7% treatment vs 50.6% control); total college credits earned (27.5 treatment vs 27.1 control); and percent of students attaining a credential or transferring to 4-year institution (23.4% treatment vs 23.7% control).
- Of the above, only the effect on completion of college-level English was statistically significant.
- In exploratory analyses, the study found suggestive positive effects among the subgroup of students that MMA bumped up to college-level courses. These subgroup findings may warrant examination in future research.
Comment:
- We’ve also reviewed a high-quality RCT of an MMA system in Minnesota and Wisconsin community colleges, which – like the New York RCT – found little impact on student success at the 4.5-year follow-up.
Click or tap a highlight to see No-Spin’s comment