R2DE: a NLP approach to estimating IRT parameters of newly generated questions
Table 10. Results of the test on performance prediction, using only the questions in DSTEST.
['Approach', 'Acc.', 'Correct Answers Prec.', 'Correct Answers Rec.', 'Wrong Answers Prec.', 'Wrong Answers Rec.']
Two different tests are carried out for performance prediction. First, we filter the validation dataset (DSVAL) in order to keep only the test questions (i.e., the ones stored in DSTEST). Then, only the test questions are used both for measuring the accuracy of the prediction and for updating the estimation of the skill level at each step. The table shows that the latent traits estimated with R2DE lead, for most of the metrics, to values that are close to the ones obtained with ground truth latent traits and generally outperform majority prediction. This means that R2DE is able to compute the latent traits from the question text with a performance similar to IRT that, instead, is based on hundreds of interactions.