AMALGUM – A Free, Balanced, Multilayer English Web Corpus
Table 2: Tokenization Performance.
['[EMPTY]', 'P', 'R', 'F1']
Although tokenization is marginally better on GUM test for the StanfordNLP GUM model (+0.03%), the results on the AMALGUM sample (+0.73%), as well as qualitative inspection of some of the frequent patterns in the complete AMALGUM data, suggest that our own tokenizer is the better choice overall for this corpus.