Controlled Experiments for Word Embeddings
Table 1: Number of words, by frequency band, as observed in the unmodified corpus.
['frequency band', '# words', 'example words']
Our training data is built from the Wikipedia data dump from October 2013. To remove the bulk of robot-generated pages from the training data, only pages with at least 20 monthly page views are retained. Punctuation marks and numbers were removed from the pages and all words were lower-cased. For recognisability, the pseudowords inserted into the corpus are upper-cased.