All Reports

University of New South Wales Study Shows ChatGPT Increases Student Coding Performance While Reducing Knowledge Retention

Bot Mutiny |

Summary

Researchers found that computer science students using ChatGPT-4.5 scored 20% higher on coding tasks but performed significantly worse on recall tests administered 48 hours later.

Researchers at the University of New South Wales report a gap between how well students completed programming tasks with ChatGPT and how much they recalled afterwards. The study, Your Programming Students' Cognition with ChatGPT: Higher Performance, Lower Retention, and Reduced Ownership, was posted to arXiv on September 18, 2026. It describes a controlled between-subjects experiment with 59 undergraduate computer science students at a single university. Fifty-five were retained for analysis. Participants completed three introductory C programming tasks with access to either ChatGPT-4.5 or conventional web search without generative AI. The authors present it as evidence of a "dissociation" between short-term task success and recall 48 hours later.

Higher Scores, Lower Recall

The AI-assisted group had higher coding scores than the group using web search, 89% against 69%. Retention was measured with cued recall questions asked immediately after the tasks and again 48 hours later. The AI-assisted group scored 41% on the immediate recall test against 53% for the web search group, and 39% against 52% after 48 hours. The authors report no significant difference between the groups in how much recall was lost over the 48 hours.

Mental Effort and Ownership

Self-reported mental effort increased less across tasks in the ChatGPT condition (Holm-adjusted p = .047). Students were also asked what percentage of the submitted code they considered their own. Students using ChatGPT attributed 45% of the code to themselves. The web search group attributed 81%.

Physiological Data

The researchers also recorded pupillary responses and heart rate variability. The confirmatory physiological tests did not detect significant differences in trajectories between the conditions. The authors write that substantial data loss limits their interpretation, and that the heart rate analysis retained only a minority of task intervals after quality checks. They say the heart rate result should be read as inconclusive rather than as evidence of no group difference.

Limitations and Conclusions

The authors list several limitations. Participants self-selected and all came from a single first-year cohort, which could introduce self-selection bias, and the results may not be generalisable to all students. The study did not control how students used ChatGPT. In the AI-assisted group of 29, 15 copied and pasted the instructions or starter code into ChatGPT and accepted the solution, while 14 wrote code themselves and used it only for validation or debugging. The authors also note that participants may have been less engaged than in a graded activity.

The authors conclude that educators who use assessment performance as a proxy for knowledge, and students who get high marks on those assessments, can be lulled into an "illusion of learning." They say the findings motivate assessment practices and AI learning tools that require students to explain, retrieve, and contribute to the work they submit.

Updated October 2, 2026 after a source verification pass.