MatInf: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization
Table 6: Experimental results of baseline methods on MatInf-C in terms of accuracy. †: Character-based models.
['Method', 'AGE', 'TOPIC']
Classification. On the tougher MatInf-C-Topic, language models prominently outperform other baselines. Among non-LM neural networks, DPCNN dpcnn, which has the deepest architecture and the most parameters, outperforms other baselines with a considerable margin. On MatInf-C-Age, which is a smaller dataset with fewer classes, DPCNN outperforms all other baselines including language models with an accuracy of 91.02. To analyze, this task has fewer training samples, which is in favor of a model with moderate parameter numbers instead of huge parameter numbers as in language models. Also, the task is relatively easier due to the class number, which makes the advantage of language models more trivial. For the multi-task baseline, MTF-S2S shows a satisfying performance on both MatInf-C-Age and MatInf-C-Topic, outperforming the same model which is only trained on the single task by 0.14 and 0.19 in terms of accuracy. Notably, BERT-of-Theseus theseus has a satisfying performance compressing the fine-tuned BERT to smaller models.