Human acceptability judgements for extractive sentence compression
Table 3: Test set accuracy, Fleiss’ κ and ROC AUC scores for six models, trained on the single-prune dataset (§4), as well as scores for a model trained on the CoLA dataset Warstadt et al. (2018). The simplest model uses only language modeling (LM) features. We add dependency type (+ dependencies) and worker ID (+ worker IDs) information to this simple model. We also remove dependency information (- dependencies) and worker information (- worker ID) from the full model. The full model achieves the highest test set AUC; p values beside each smaller AUC score show the probability that the full model’s gains over the smaller AUC score occurs by chance. We also compute κ for each model by calculating the observed and pairwise agreement rates Fleiss (1971) for judgements submitted by the crowdworker and “judgements” submitted by the model. Models which can account for worker effects achieve higher accuracies than the observed agreement rate among workers (0.636∗), leading to higher κ than for worker–worker pairs.
['Model', 'Hard classification ( [ITALIC] t=0.5) Accuracy', 'Hard classification ( [ITALIC] t=0.5) Fleiss  [ITALIC] κ', 'Ranking ROC AUC ( [ITALIC] p)']
We use Norm LP as a part of two features in our approach. One real-valued feature records the probability of a compression computed by Norm LP(c). Another binary feature computes Norm LP(s) - Norm LP(c) > 0. The test set performance of these language model The appendix further describes our implementation of Norm LP. Lau et al. We compute the test set accuracy of each approach in predicting binary judgements from individual workers, which allows for comparison with agreement rates between workers. ROC AUC thus measures how well predicted probabilities of binary positive judgements correlate with human perceptions of well-formedness. Other work which solicits gradient-based judgements instead of binary judgements evaluates with Pearson correlation Lau et al. Our full model also achieves a higher AUC than approaches that remove features from the model. We use bootstrap sampling Berg-Kirkpatrick et al. We also compare our full model to a baseline neural acceptability predictor trained on CoLA Warstadt et al. Using a pretrained model, we predict the probability that each source sentence and each compression is well formed, denoted CoLA(s) and CoLA(c). We use these predictions to define four features: CoLA(c), log CoLA(c), CoLA(s) - CoLA(c), and log CoLA(s) - log CoLA(c). CoLA’s performance for extractive compression results warrants future examination: large corpora designed for neural methods sometimes contain limitations which are not initially understood Chen et al.