Neural Network Acceptability Judgments
Table 6: MCC results for specific phenomena. Emb. is model embedding style; Enc. is model encoder training, Class. is model classifier training. R/F is real/fake, ELMo-St. is ELMo-style, and CoLA-Th. is threshold tuning on CoLA. LSTM LM SLOR/WLPM is the LM with Lau et al. metrics Word LP Min-1.
['[BOLD] Model', '[BOLD] Emb.', '[BOLD] Enc.', '[BOLD] Class.', '[BOLD] SVO', '[BOLD] Wh', '[BOLD] Causative', '[BOLD] SV Agr.', '[BOLD] Reflexive']
Our models’ performance on these test sets is mixed. They make some systematic acceptability judgments that reflect correct grammatical generalizations. Some models are very effective at judging violations in gross word order ( The pooling classifier with GloVe embeddings achieves near perfect correlation, suggesting that it systematically uses gross word order. However, the remaining tests yield much poorer performance.