QuesNet: A Unified Representation for Heterogeneous Test Questions
Table 4. Ablation experiments.
['Methods', 'Knowledge mapping ACC', 'Knowledge mapping Precision', 'Knowledge mapping Recall', 'Knowledge mapping F-1', 'Difficulty estimation MAE', 'Difficulty estimation RMSE', 'Difficulty estimation DOA', 'Difficulty estimation PCC', 'Student performance prediction MAE', 'Student performance prediction RMSE', 'Student performance prediction ACC', 'Student performance prediction AUC']
In this section, we conduct some ablation experiments to further show how each part of our method affect final results. Specifically: QN-T, QN-I, QN-M refer to QuesNet with only text, image or side information is used in pre-training process, respectively. QN-TI, QN-TM, QN-IM refer to combinations of different kinds of input, i.e. text and image, text and side information, and image and side information, respectively. And finally, QN-L refers to QuesNet that only includes low-lever pre-training objective (holed language model), and QN-H refers to QuesNet that only includes high-level domain objective. First, the more information a model incorporates, the better the performance, which agrees with the intuition. Second, if we focus on comparison between QN-H and QN-L, we will notice that they gain different effects on different tasks. On tasks more focusing on lower level features like knowledge mapping, QN-L outperforms QN-H slightly, while on other more domain-oriented high-level tasks (difficulty estimation and student performance prediction), QN-H performs a little better. This clearly demonstrates different aspects the two objectives focus on, and with full QuesNet outperforms both QN-L and QN-H. we know that QuesNet is able to take account of both aspects of these objectives, and build an understanding at both low linguistic level and high logic level. Third, we notice that QN-T has better performance than QN-I, which is better than QN-M, indicating that text carries most information in a question, then images, then side information. But as omitting either of the input kind will cause a performance loss, all of the information in heterogeneous inputs is essential to a good question understanding.