Leveraging large language models to identify and classify student errors in higher education mathematics using the STACK dataset

STACK is an interactive digital teaching tool for STEM education. While it has helped address challenges such as large class sizes in institutions of higher learning in Africa, it lacks adaptability to individual student needs. The integration of machine learning (ML) techniques in STACK to support adaptive mathematics learning in universities has gained our attention. However, effectively identifying the types of errors learners make is crucial for developing a recommendation system that enhances both emotional and behavioral student engagement. This study explores how integrating Large Language Models (LLMs), specifically OpenAI’s GPT-4o, with Newman’s Error Analysis (NEA) framework can enhance STACK’s diagnostic capabilities for personalized learning in mathematics. We analyzed 2,834 incorrect responses from 618 first-year students at Masinde Muliro University of Science and Technology (MMUST), Kenya, in the topic “Plane Curves and Parametric Equations”. GPT-4o was prompted to classify student errors based on the five NEA categories: reading, comprehension, transformation, process skill, and encoding. Validation against expert judgments demonstrated substantial agreement, particularly in transformation and comprehension errors, with Cohen’s Kappa values exceeding 0.70. The model achieved high micro-averaged F1 (87.98%) and macro-averaged F1 (85.83%) scores, confirming reliable multi-label classification. Further, machine learning models were used to predict student performance levels (low, medium, high) using error patterns and grades. Random Forest achieved the highest accuracy (99.65%), providing interpretable insights into the most predictive error types. Distinct error distributions across performance groups were used to recommend targeted instructional strategies. Experts rated GPT-4o explanations as 100% clear, accurate, and pedagogically useful. Overall, this study demonstrates the potential of integrating LLMs with STACK to automate error diagnosis and provide detailed pedagogical feedback. The findings will support the development of real-time adaptive features in STACK, including AI-powered agents for personalized recommendations, thereby enhancing instructional quality and scalability in mathematics education.