WikiAtomicEdits: A Multilingual Corpus of Wikipedia Editsfor Modeling Language and Discourse
Table 9: Relationship between model accuracy and insertion type, based on a sample of 50 correct (✓) and 50 incorrect (✗) predictions from each model. Base frequency is shown for reference and is based on our analysis from §5.1. The General LM shows a bias in accuracy by insertion type. This bias is not observed for the discriminative model.
['[EMPTY]', 'Base Freq.', 'General LM', 'General LM', 'Discr. Model', 'Discr. Model']
More interesting than raw performance is the difference in the types of errors that the models make. For each model, we take a random sample of 50 examples on which the model made a correct prediction and 50 examples on which the model made an incorrect prediction. Specifically, the General LM appears to be especially good at predicting location for fluency/discourse edits, and especially poor at predicting the location of refinement edits. In contrast, we do not see any significant bias in the errors made by the discriminative model compared to its correct predictions (p=0.23). We interpret this as evidence that the insertion data captures some semantic signal that is not readily gleaned from raw text corpora.