dAFM: Fusing Psychometric and Connectionist Modeling for Q-matrix Refinement

We introduce a model which combines principles from psychometric and connectionist paradigms to allow direct Q-matrix refinement via backpropagation. We call this model dAFM, based on augmentation of the original Additive Factors Model (AFM), whose calculations and constraints we show can be exactly replicated within the framework of neural networks. In order to parameterize the Q-matrix definition in the model, the associations between questions and knowledge components (KC) need to be represented by adjustable weights. Furthermore, student KC opportunity counts, instead of serving as fixed inputs, need to be calculated dynamically as the Q-matrix changes during training. We describe our solutions to these two modeling challenges and evaluate several variants of our fully realized model on datasets from the Cognitive Tutor and ASSISTments. We compare learning the Q-matrix from scratch vs. refining an expert specified KC model and evaluate various procedures for refinement. In our quantitative predictive analysis, we find that dAFM learns a better generalizing Q-matrix than the original expert model in all our primary datasets. Using a development set, we also find that the dAFM Q-matrix is superior to KC representations extracted from trained Deep Knowledge Tracing and skip-gram models. Examples are shown of questions whose fit was improved by dAFM with depictions of their original and refined KC associations. We consistently find in our experiments that our dAFM variant which attempted to learn the Q-matrix from scratch underperformed models which started with an expert defined Q-matrix that was then refined. This observation continues a theme in EDM of utility found in the enduring value of expert domain knowledge enhanced through data-driven refinement.