Neural Network Acceptability Judgments
Table 4: Results for acceptability classification on the CoLA test set. The first group is the CBOW baseline. The second group is the LSTM and n-gram LMs with Lau et al.’s metrics. The third group is pooling classifiers trained end-to-end on the real/fake objective. The fourth group is pooling classifiers with training on CoLA, mostly with encoders transferred from real/fake classifiers. The fifth group is the small human evaluations (Section 3). CoLA-Thresh. is threshold tuning on CoLA, and WLPM is Lau et al.’s Word LogProb Min-1 metric.
['[BOLD] Model', '[BOLD] Embeddings', '[BOLD] 0.5inEncoder Training', '[BOLD] 0.5inClassifier Training', '[BOLD] In-domain  [BOLD] Acc.', '[BOLD] In-domain  [BOLD] MCC', '[BOLD] Out-of-domain  [BOLD] Acc.', '[BOLD] Out-of-domain  [BOLD] MCC']
Average annotator agreement with CoLA is 86.1%, and average Matthews Correlation Coefficient (MCC) Selecting the majority decision from our annotators gives us a rough upper bound on human performance. These judgments agreed with CoLA’s ratings on 87% of sentences with an MCC of 0.713. In other words, 13% of the labels in CoLA contradict the observed majority judgment. First, we train classifiers end-to-end on the real/fake task, varying the style of word embedding. The classifier portion consists only of a single softmax layer. We evaluate these classifiers on CoLA without CoLA training. The best model overall is the real/fake model with ELMo-style embeddings. It achieves the highest MCC and accuracy both in-domain and out-of-domain by a large margin, outperforming even the models with access to GloVe.