Neural Network Acceptability Judgments
Table 5: Results for 5 different splits of CoLA and the original split into in-domain and out-of-domain. All results are averages over 20 restarts. Out N is the number of out-of-domain sentences. Sources are abbreviated by authors’ last initial and year; full citations for each source are shown in Table 2.
['[BOLD] Split', '[BOLD] In-Domain Acc.', '[BOLD] In-Domain MCC', '[BOLD] Out-of-Domain Acc.', '[BOLD] Out-of-Domain MCC', '[BOLD] Overall Acc.', '[BOLD] Overall MCC', '[BOLD] Out Sources', '[BOLD] Out N']
We conclude that the domain difference between two samples of sources in CoLA is generally a meaningful one for these models. This is especially so for the original split, where average in-domain MCC is 0.125 greater than out-of-domain MCC, close to the maximum observed difference of 0.162. By contrast, in one case average out-of-domain performance was actually better. This tells us that the particular nature of the sources in each domain has a large effect on what our models learn.