WikiAtomicEdits: A Multilingual Corpus of Wikipedia Editsfor Modeling Language and Discourse
Table 2: The number of instances (in millions) of atomic insertions/deletions for each language.
['Language', 'Ins', 'Del', 'Total']
Given n sentences in one snapshot (“base”) and m sentences in a subsequent one (“edited”), we assume that most edits are local and restrict our attention to a fixed-size window. For each sentence si in the base snapshot, we compute pairwise BLEU scores Papineni et al. We consider the sentence with the highest BLEU score in this window as a candidate. For each article, we run this algorithm over the most recent 100,000 snapshots as of February 2018. We extract edits for 8 languages. We now turn our attention to exploring the language in the corpus itself. and 2) what can we learn about language by observing the editing process that we cannot readily learn by observing only raw text? Here, we explore these questions from a corpus linguistics perspective.