Do Neural Language Representations Learn Physical Commonsense?
Table 2: Macro F1 scores per category (object, property, affordance) and micro F1 score (μF1) on both the abstract and situated test sets. Highest model values are bolded. Statistical significance (sig) is calculated with McNemar’s test, comparing the best-scoring model (by μF1, denoted ←) with each other model. Stratified p-values are shown, with * for p<0.05, ** for p<0.01, and *** for p<0.001. Human performance is estimated by 50 expert-annotated random samples from the test set (no McNemar’s test).
['[EMPTY]', '[BOLD] Abstract O ⟷ P', '[BOLD] Abstract O ⟷ P', '[BOLD] Abstract O ⟷ P', '[BOLD] Abstract O ⟷ P', '[BOLD] Situated O ⟷ P', '[BOLD] Situated O ⟷ P', '[BOLD] Situated O ⟷ P', '[BOLD] Situated O ⟷ P', '[BOLD] Situated O ⟷ A', '[BOLD] Situated O ⟷ A', '[BOLD] Situated O ⟷ A', '[BOLD] Situated O ⟷ A', '[BOLD] Situated A ⟷ P', '[BOLD] Situated A ⟷ P', '[BOLD] Situated A ⟷ P', '[BOLD] Situated A ⟷ P']
Models fare slightly worse on the situated O⟷P task, with the best performance below 0.60 F1. This effect is consistent in the human expert scores as well. Though this dataset is larger, the introduction of context allows for greater variance in the properties of an object.