EMNLP’19 “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding
Table 2: Summary of the performances for different baselines. All numbers are in percentages.
['System', '[ITALIC] F1', '[ITALIC] EM']
We can confirm the good quality of this dataset based on the high performance of human annotators. ELMo and BERT improve naive baselines by a large margin, indicating that a notable amount of commonsense knowledge has been acquired via pre-training. However, even BERT still falls far behind human performance, indicating the need of further research. For an ideal system, the gap between EM and F1 should be small (humans only drop 11.3%.) However, all other systems drop by almost 30% from F1 to EM, possibly another piece of evidence that they only associate surface forms instead of using one representation for temporal commonsense to classify all candidates. This is expected because commonsense is what most people agree on, so any single human could disagree with the gold labels in McTaco.