Rethinking Knowledge Tracing for AI-Assisted Classroom Tutoring

Teaching is a fundamental human activity that relies on a teacher’s ability to monitor a student’s progress and adjust lessons to meet their specific needs. To make learning more engaging and effective, personalized learning is crucial. A key part of personalized learning is “knowledge tracing” which means figuring out how much a student understands different knowledge concepts as they learn.

Knowledge tracing is an approach used in education to model, track and predict a student’s progression of specific skills and knowledge concepts over time. At its core, knowledge tracing uses data from a student’s past performance from quizzes to estimate the likelihood that they have mastered certain skills and predict their performance on future tasks. It is the modelling of a learner’s knowledge state from observed learning interactions, quiz attempts, practice logs, or problem-solving responses (Abdelrahman et al., 2023). To understand a student’s learning progress, a single aggregate score is rarely enough. Skill-level tracing allows the instructor to plan the next session more precisely. It can be based on which skill needs reteaching, which students need guided practice, which students are ready for application, and which quiz items should be selected next.

Contemporary classroom assessment, however, introduces a new measurement problem. Students increasingly complete quizzes in open-book and AI-supported conditions. In such settings, a correct answer may represent independent mastery, but it may also represent successful lookup, recognition, AI-assisted reasoning, or copying of a generated answer. Recent work on AI-supported assessment has argued that generative AI requires assessment design to distinguish between answer production, verification, judgement, and responsible tool use (Perkins et al., 2023; Chan, 2023). This creates a measurement problem for knowledge tracing models because the observed response may no longer represent only the learner’s latent mastery. A correct answer may indicate mastery, but it may also indicate successful retrieval, AI prompting, recognition, or assisted reasoning. This article therefore suggests that knowledge tracing remains useful for classroom tutoring, but its parameters and outputs must be reinterpreted when assessments are AI-supported.

Bayesian Knowledge Tracing (BKT) Model

The Bayesian Knowledge Tracing (BKT) model is an analytic technique for estimating learners’ mastery of skills after exposure to learning materials and guidance. It is widely used in its basic form despite the existence of more complex models that account for difficulty, because of its useful skill level diagnostic abilities for teachers (Pardos & Heffernan, 2011). It is a traditional knowledge tracing method that simplifies the problem by assuming that the knowledge state is a binary random variable or that each question involves only one knowledge component. Bayesian Knowledge Tracing (BKT) models a learner’s mastery as a latent, skill-specific probability that is updated every time the learner answers a question. In its standard form, it assumes that knowledge at the skill level:

1. Is binary and latent (learned or unlearned),
2. Evolves as a simple hidden Markov process, and
3. Slips and guesses are stationary within a skill.

Items are mapped to skills in a Q-matrix; when a learner answers an item, the model updates all tagged skills. This is usually done by looking at their past attempts on exercises and how those exercises relate to specific knowledge concepts. The Q-matrix is like a map that shows which knowledge concepts are covered by each exercise (Wang et al., 2022).

Here’s, how to approach and visualise a Q-matrix.

Q-matrix for knowledge tracing (Wang et al., 2022)

Each row in the above Q-matrix indicates which skills are required for a particular question. The Q-matrix is often expert-defined, but recent models and research seek to learn or refine it automatically for interpretability and improved performance (Jia et al., 2023). BKT models have 4 parameters:

P(S) & P(G) are emission probabilities. Slip probability P(S) is the chance of making a mistake with a known skill and the guess probability P(G) is the probability of a correct response when the learner has not mastered the skill. Transition or learning probabilities are described by P(L) of a skill. In BKT, the probability of forgetting a previously known skill P(F) is assumed to be zero. The learning task in BKT involves estimating these parameters based on a sequence of observations. Because its inference proceeds one skill at a time via a first-order Hidden Markov Model, BKT does not require large training corpora that attention-based Deep Knowledge Tracing (DKT) models need (Abdelrahman et al., 2023).

BKT offers a straightforward, interpretable framework by estimating the probabilities of learning, guessing, and slipping on a skill-by-skill basis. To see how BKT works end-to-end, consider a single skill, “Ratios” with parameters seeded from historical data or expert priors as pL = 0.12, pS = 0.08, pG = 0.20, pF = 0. Suppose a learner answers four Ratio problems in sequence: correct, incorrect, correct, correct.

If the response is correct (X_t=1):

If the response is incorrect (X_t=0):

Transition (learning pL; pF=0):

The model’s predicted probability of a correct response before seeing it is:

After the first correct response, the posterior mastery rises sharply because a correct answer is much more likely if the learner knows the skill than if they guessed; after applying the learning step, mastery reaches to 0.747. The second response is incorrect, so the posterior drops substantially; after adding the learning step, mastery is about 0.321. A third correct pushes it back up to about 0.722, and a fourth correct moves it to about 0.932.

This trajectory illustrates two characteristic features of BKT.

1. Mastery is not strictly monotonic; it can fall after an error because the model updates its belief about whether the learner truly knows the skill.
2. Even after an incorrect answer, the learn rate gives a small “practice bump,” reflecting that attempting an item can still improve knowledge.

For teaching purposes, low mastery values suggest reteaching, mid-range values suggest guided practice, consolidation-range values suggest reinforcement and mixed practice, and high values suggest readiness for advanced or application tasks.

The standard BKT model is useful, but direct classroom adoption requires modification. A classroom tutor usually has fewer observations than a large platform, uneven coverage across skills, incomplete baseline data, and assessment conditions where students may use notes or AI tools. Because the BKT model assumes that all students have the same prior knowledge and learning rate for any given skill, it can also underestimate the learning performance of above-average students and overestimate the performance of below-average students.

Contamination of mastery due to AI assistance

BKT’s explicit and interpretable parameters align closely with educational theories, rendering the model transparent and accessible to practitioners and educators. Standard BKT assumes that a correct answer reflects the student’s own latent knowledge state, filtered only through a small, mechanically bounded chance of guessing, p(G) (Corbett & Anderson, 1994). AI-assisted responses break the BKT’s core assumptions that correctness is generated by the student’s own latent knowledge state. Once a student can look something up or ask an AI system, a correct answer might reflect successful information retrieval rather than mastery and BKT has no built-in way to tell the difference. Therefore, the main challenge is not that students can get answers correct. The challenge is that correctness becomes ambiguous evidence.

Recent discussions of use of AI in assessment also argue that assessment design should clarify permitted AI use and should align evidence of performance with the intended learning outcome (Chan, 2023; Perkins et al., 2023). In the BKT model, p(G) is meant to represent random guessing by an unlearned student that is typically a small, mechanically bounded number such as 1/n for multiple choice questions. AI-support inflates the true guess rate for test takers who haven’t mastered the skill, because they now have a systematic (not random) way to get the right answer without knowing it. Any attempt to estimate the additional parameter to describe AI use from a classroom dataset will simply overfit, producing an unstable or degenerate result rather than a meaningful correction.

This creates a practical tension. On one hand, ignoring AI entirely means accepting a knowledge trace that may substantially misrepresent individual mastery. On the other hand, attempting to model these effects with the same statistical machinery used at platform scale is not viable with classroom-sized data.

The core assumption being violated is that the correct answer which reflects the student’s own knowledge state is via p(S) (Slip) and p(G) (Guess). When a student consults an AI system, a correct answer may instead reflect successful retrieval. It is functionally a much higher and non-random “guess” rate that the standard model has no parameter to represent (Corbett & Anderson, 1994). The problem is that a high correct-rate on an item is compatible with two very different underlying stories, and nothing in the response tells which one is true:

1. Genuine mastery: the class understands the skill well, so p(L) should be high.
2. AI-driven inflation: many students don’t understand the skill but got it right anyway via AI, so p(G) should be high.

In a large dataset, this ambiguity can be broken using sequential evidence e.g, a student’s performance on earlier, easier items in the same skill can help disambiguate whether later high performance reflects real learning or something else. A single classroom quiz typically doesn’t have that kind of longitudinal signal within one skill, so the ambiguity mostly stays unresolved. Thus, fitting a single p(G) from the class-wide pass rate averages over students who did and didn’t use AI. Even if the average shift is roughly in the right direction, individual students who solved the item honestly will have their true mastery systematically underestimated, while students who used AI will have their mastery overestimated less than they should.

Discrimination-index method for informing the fixed guess value

In classical test analysis, item p-value refers to the proportion of students answering an item correctly. In this article, Item_P is interpreted as observed answerability under the actual quiz condition. In AI-supported settings, difficulty should be treated as observed struggle under assisted conditions, not as intrinsic cognitive difficulty.

The critical design choice would be that the AI guess parameter, p(G_open), should be fixed at an instructor-specified value. An instructor who knows, for example, that a capable AI tool can solve a given item with high reliability regardless of the student’s understanding can set p(G_open) accordingly (e.g., 0.7–0.9) as a modeling assumption rather than a fitted quantity. This substitutes expert judgment for statistical power that the sample does not have.

A more diagnostic quantity is the item’s discrimination index, a classical test theory statistic that has been used specifically because it can be computed by counting rather than by model fitting, making it appropriate for the sample sizes typical of a single classroom (Kelley, 1939). This reasoning is analogous to the discrimination parameter aⱼ in the 3PL model. An item that validly measures mastery should yield higher rates of correct responses among higher-ability students. By contrast, lower-ability students would be expected to answer such an item correctly less often. The method proceeds as follows:

• Compute a rest-score per student. For each student, sum correct responses across all other items measuring the same skill, excluding the item under review. This serves as an ability proxy that is independent of the item being diagnosed.
• Split students into upper and lower groups by rest-score. The classical convention uses the top and bottom 27% of the rest-score distribution, a cutoff chosen because it maximizes group separation while retaining enough students per group for a stable comparison (Kelley, 1939). For classes small enough that a 27% split leaves very few students per group, a median split (top half vs. bottom half) is an acceptable substitute.
• Calculate the discrimination index, D, as the difference in proportion correct between the two groups:

D = (proportion correct, upper group) − (proportion correct, lower group)

• Interpret D as a triage signal, not as a statistical test. Conventional thresholds treat D ≥ 0.30 as good discrimination. A value between 0.10 and 0.29 is usually considered marginal. A value below 0.10, or a negative value, suggests poor discrimination and should be reviewed. The strongest warning sign is a low or negative D combined with a high overall item pass rate. In this case, the item is easy for most students but does not separate stronger students from weaker students. This pattern is not consistent with mastery-driven correctness. A genuinely easy item should still show at least a small positive difference between the two groups. By contrast, correctness driven by shared external resource use tends to reduce or remove that difference.

A high pass rate combined with low or negative discrimination is a warning signal. It suggests that the item is easy for most students but does not separate stronger from weaker students. This pattern may arise from AI assistance, but it may also arise from poor wording, weak skill tagging, overly simple content, or genuine uniform mastery. Therefore, discrimination should not be used as proof of AI use. It should be used as a prompt for instructor review and possible adjustment of the correct-without-mastery assumption.

This diagnostic should support the instructor’s judgment when setting p(G_open). It helps identify which items need closer review and may require a higher assumed guess rate. However, it does not estimate that rate directly. It also should not replace the fixed-parameter approach which can also become noisy when group sizes are very small. In those cases, one student’s response can noticeably change the index. Therefore, D should be treated as a prompt for instructor follow-up, not as a definitive classification of an item.

AI-assisted responses violate the p(Guess) assumptions of the standard BKT model and therefore require remedies. It can be best handled by re-specifying the emission model which is splitting guess/slip parameters by resource condition and fixing, rather than estimating, the contaminated parameter. BKT should serve as the first interpretable benchmark. Once multiple quizzes, response times, item metadata, AI-use disclosures, revision logs, and explanation scores are available, more flexible models can be evaluated against BKT. These recommendations can allow an instructor to build a more defensible classroom-scale knowledge tracing tool without importing solutions that depend on the scale of data available only to large learning platforms.

Reference

• Abdelrahman, G., Wang, Q., & Nunes, B. (2023). Knowledge Tracing: A Survey. ACM Comput. Surv., 55(11), 224:1-224:37. https://doi.org/10.1145/3569576
• Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20(1), 38. https://doi.org/10.1186/s41239-023-00408-3
• Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. https://doi.org/10.1007/BF01099821
• Jia, Z., Su, W., Liu, J., & Yue, W. (2023). Attentive Q-Matrix Learning for Knowledge Tracing (arXiv:2304.08168). arXiv. https://doi.org/10.48550/arXiv.2304.08168
• Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17–24. https://doi.org/10.1037/h0057123
• Pardos, Z. A., & Heffernan, N. T. (2011). KT-IDEM: Introducing item difficulty to the knowledge tracing model. Proceedings of the 19th International Conference on User Modeling, Adaption, and Personalization, UMAP’11, 243–254.
• Perkins, M., Furze, L., Roe, J., & MacVaugh, J. (2023). The AI Assessment Scale (AIAS): A Framework for Ethical Integration of Generative AI in Educational Assessment. https://doi.org/10.48550/ARXIV.2312.07086
• Wang, W., Ma, H., Zhao, Y., Li, Z., & He, X. (2022). Tracking knowledge proficiency of students with calibrated Q-matrix. Expert Systems with Applications, 192, 116454. https://doi.org/10.1016/j.eswa.2021.116454