QuesNet: A Unified Representation for Heterogeneous Test Questions
Table 3. Performance of comparison methods on different tasks.
['Methods', 'Knowledge mapping ACC', 'Knowledge mapping Precision', 'Knowledge mapping Recall', 'Knowledge mapping F-1', 'Difficulty estimation MAE', 'Difficulty estimation RMSE', 'Difficulty estimation DOA', 'Difficulty estimation PCC', 'Student performance prediction MAE', 'Student performance prediction RMSE', 'Student performance prediction ACC', 'Student performance prediction AUC']
We can easily see that pre-trained methods is able to boost the performance on each task, among which QuesNet has the best performance on almost all metrics, no matter what the size of each task is. This proves that QuesNet gains a better understanding of questions and is transfered more efficiently from large unlabeled corpus to small labeled datasets. However, there are more to be explained in this table. First, models that support heterogeneous input have superior results over similar structures without heterogeneous input, which proves that it is crucial to handle heterogeneous inputs when understanding questions. Second, as all methods are adjusted to contain similar amount of parameters, QuesNet turns out to be the most efficient one. Third, the result of the Transformer based methods is slightly lower than other LSTM based methods. This is probably because of the low sample efficiency of masked language model pre-training strategy used in BERT, while bi-directional language model in ELMo performs better, and our novel holed language model used in QuesNet outperforms both of them.