Quantity doesn’t buy quality syntax with neural language models
Table 1: Strength of evidence for improvements in agreement prediction accuracy as a result of increasing corpus size averaging across layer size (left) or layer size averaging across corpus size (right), as quantified by Bayes factors. Boldfaced Bayes factors indicate strong evidence of improvement.
['Corpus size 2M → 10M', 'Corpus size  [BOLD] 5508.8', 'Layer size 100 → 200', 'Layer size  [BOLD] 768.5']
Increasing model size improved syntactic prediction accuracy up to 400 units per layer; further increases in model size had no effect (see Increasing the amount of training data impacted accuracy in an inconsistent way. Training on 10M tokens produced general improvements across all constructions compared to 2M tokens. Doubling the corpus to 20M did not affect accuracy, but doubling it again to 40M did. There was no evidence of further improvement between 40M and 80M words.