KaLM at SemEval-2020 Task 4: Knowledge-aware Language Models for Comprehension And Generation
Table 3: Model performance on the official subtask C development set and test set. Multi-target: the same input has multiple different targets. Extra Words: extra words indicating given statement does not make sense. Reasonable Statement: the corresponding reasonable statement. Wiktionary: the obtained evidence from wiktionary
['[EMPTY]', 'Dev BLEU', 'Test BLEU']
Compare to the baseline where we simply use the first referential answer of each example in training data as the target of the model output, the Multi-target method provides a larger amount of training data and thus helps the model get better performance. We hypothesize the degradation is because the complexity of the input sentence increases as we add extra material. When Extra Words, Reasonable Statement, and Wiktionary are all added, the benefits outweigh the disadvantages they bring. Therefore, we use all of them in the final submission of subtask C and achieve the BLEU score of 18.5 (3rd place) and human evaluation score of 2.08 (2nd place), which obtains a 0.14 gain over 3rd place and only 0.02 less than 1st place. Note that we might achieve a better human evaluation score accordingly. There are two reasons for that. Firstly, we optimize our evidence-searching approach after the competition, removing a big proportion of noisy data. Secondly, we observe that during the training process, the model performs well at the beginning but turns to mess later. Thus it’s difficult to choose the best model during the training process when we cannot evaluate it on the test set. When the competition has ended, however, we can evaluate our models and choose the best one during the training process.