How to Generate a Good Word Embedding?
Table 9: PGRs of avg task when trained on mixed corpora. Each column corresponds to the percentage of tokens we use in IMDB corpus, and each row corresponds to the percentage of tokens we use in the 13M-token subset of W&N corpus. For example, the entry with value 68 in the third row and the first column means we combine 20% of the IMDB corpus with 40% of the 13M subset of W&N corpus to train a word embedding, and obtain a PGR of 68 on avg task.
['W&NIMDB', '20%', '40%', '60%', '80%', '100%']
In previous sections, the experiment results demonstrated more in-domain training data would benefit to training a precise model. However, for a specific task, it may be impossible to obtain sufficient in-domain data. Increasing more training data may introduce more out-domain data and may contradict the demand of in-domain. In this case, should we keep the corpus pure or add the out-domain corpus? We further design an experiment to answer this question. In this experiment, we use the 13M-token IMDB corpus and a subset of W&N corpus with 13M tokens. The subsets of the both corpora are mixed to train word embedding. For each column in the table, the elements are listed in ascending order of corpus size, while in descending order of the in-domain purity. The experiment results demonstrate that no matter in which scale the IMDB corpus is, adding the W&N corpus consistently decrease the performance. The pure IMDB corpus is better than the larger mixed corpus.