Some of Them Can be Guessed!Exploring the Effect of Linguistic Context in Predicting Quantifiers
Table 2: Accuracy of models and humans. Values in bold are the highest in the column. *Note that due to an imperfect balancing of data, chance level for humans (computed as majority class) is 0.124.
['[EMPTY]', '1-Sent  [ITALIC] val', '1-Sent  [ITALIC] test', '3-Sent  [ITALIC] val', '3-Sent  [ITALIC] test']
We have three main results. (1) Broader context helps humans to perform the task, but hurts model performance. This can be seen by comparing the 4-point increase of human accuracy from 1-Sent (0.22) to 3-Sent (0.26) with the generally worse performance of all models (e.g. AttCon-LSTM, from 0.34 to 0.27 in val). (2) All models are significantly better than humans in performing the task at the sentence level (1-Sent), whereas their performance is only slightly better than humans’ in 3-Sent. AttCon-LSTM, which is the best model in the former setting, achieves a significantly higher accuracy than humans’ (0.34 vs 0.22). By contrast, in 3-Sent, the performance of the best model is closer to that of humans (0.29 of Att-LSTM vs 0.26). It can be seen that LSTMs are overall the best-performing architectures, with CNN showing some potential in the handling of longer sequences (3-Sent). Compare ‘few’, ‘a few’, ‘more than half’, ‘some’, and ‘most’: while the first three are generally hard for humans but predictable by the models, the last two show the opposite pattern. Moreover, quantifiers that are guessed by humans to a larger extent in 3-Sent compared to 1-Sent, thus profiting from the broader linguistic context, do not experience the same boost with models. Human accuracy improves notably for ‘few’, ‘a few’, ‘many’, and ‘most’, while model performance on the same quantifiers does not.