R2DE: a NLP approach to estimating IRT parameters of newly generated questions
Table 11. Results of the test on performance prediction on test interactions, skill estimated using all the questions.
['Approach', 'Acc.', 'Correct Answers Prec.', 'Correct Answers Rec.', 'Wrong Answers Prec.', 'Wrong Answers Rec.']
The second test explicitly reproduces the scenario that occurs in the wild. In real assessment only some of the items are newly generated (thus requiring an estimation of the latent traits from text), most of them are already calibrated (i.e., with known latent traits). Therefore, in this second experiment on performance prediction both the test questions and the train questions of DSVAL are used. However, only the test questions are considered for evaluating the performance of the model on performance prediction; the train questions are used exclusively to update the estimated skill level of the student during the experiment. Anyway, the test questions are used also for updating the skill level estimation, as it was the case in the previous test. Majority is not reported here since it is computed as in the previous test and performed much worse than the other two approaches. Again, the latent traits estimated with R2DE proved good estimations for newly generated items. Indeed, the accuracy obtained with our model is only 1.57% lower than the accuracy obtained with IRT-estimated latent traits, which is the upper threshold.