How Furiously Can Colourless Green Ideas Sleep?Sentence Acceptability in Context
Table 3: Modelling results. Boldface indicates optimal performance in each row.
['[BOLD] Rtg', '[BOLD] Encod.', '[BOLD] Model', '[BOLD] LP', '[BOLD] MeanLP', '[BOLD] PenLP', '[BOLD] NormLP', '[BOLD] SLOR']
We use Pearson’s r to assess how well the models’ acceptability measures predict mean human acceptability ratings, following previous studies Lau et al. Recall that for each model (e.g. lstm), there are 3 variants with which we infer the sentence probability at test time. These are distinguished by whether we include no context (lstm∅), real context (lstm+), or random context (lstm−). There are also three types of human acceptability ratings (ground truth), where sentences are judged with no context, (h∅), real context (h+), and random context (h−). To get a sense of what the correlation figures indicate for these models, we compute two human performance estimates to serve as upper bounds on the accuracy of a model. The first upper bound (ub1) is the one-vs-rest annotator correlation, where we select a random annotator’s rating and compare it to the mean rating of the rest, using Pearson ’s r. We repeat this for a large number of trials (1000) to get a robust estimate of the mean correlation. ub1 can be interpreted as the average human performance working in isolation. The second upper bound (ub2) is the half-vs-half annotator correlation. For each sentence we randomly split the annotators into two groups, and compare the mean rating between groups, again using Pearson’s r and repeating it (1000) to get a robust estimate. ub2 can be taken as the average human performance working collaboratively. respectively. While this produces a cleaner set of annotations, this filtering step does (artificially) increase the human agreement or upper bound correlations. For completeness we also present upper bound variations where we do not remove the outlier ratings, and denote them as ub∅1 and ub∅2. In this setup, the one-vs-rest correlations drop to 0.62–0.66 Note that all model performances are reported based on the outlier-filtered ratings, although there are almost no perceivable changes to the performance figures when they are evaluated on the outlier-preserved ground truth.