Springer Nature has retracted a highly-cited study that claimed OpenAI’s ChatGPT significantly boosts student learning outcomes, according to Ars Technica.
The journal publisher cited "discretes" in the analysis and a lack of confidence in the paper's conclusions when withdrawing the research. The retraction comes nearly one year after the paper's initial publication in the journal Humanities & Social Sciences Communications.
Before its removal, the study gained significant traction on social media and accumulated hundreds of citations. The paper attempted to quantify the effect of ChatGPT on learning performance, perception, and higher-order thinking by analyzing 51 previous research studies.
According to the outlet, the researchers originally claimed the AI chatbot had a "large positive impact on improving learning performance" and a "moderately positive impact on enhancing learning perception."
Flaws in meta-analysis
Ben Williamson, a senior lecturer at the University of Edinburgh’s Centre for Research in Digital Education, criticized the original publication. Williamson told Ars Technica that the paper's authors made "very attention-grabbing claims about the benefits of ChatGPT on learning outcomes."
He noted that the study was widely treated as "one of the first pieces of hard, gold standard evidence" that generative AI benefits learners. However, Williamson suggested the methodology was deeply flawed.
"In some cases it appears it was synthesizing very poor quality studies, or mixing together findings from studies that simply cannot be accurately compared due to very different methods, populations, and samples," Williamson said.
He further remarked that the paper "really seemed like a paper that should not have been published in the first place."
Williamson also pointed to the improbable timeline of the research. He noted it was not feasible for dozens of high-quality studies regarding ChatGPT's impact on learning to have been produced in the short period since OpenAI released the chatbot in November 2022.
