A Systematic Study of Leveraging Subword Information for Learning Word Representations
Table 9: Results on word similarity and relatedness across languages for BPE (bpe). All scores are obtained after computing the embeddings of OOV words.
['[EMPTY]', 'simlex', 'ws', 'ws-rel', 'ws-sim', 'fs300', 'an-rel', 'an-sim', 'card']
In this supplementary material we present the full results of different configurations of our subword-informed representations and baseline models in three tasks across five languages. For details and notations of different model configurations, please refer to the original paper. Since fastText (ft) package cannot generate each subword embedding for a given word, we also implemented our own version (our_ft) within the same subword-informed training framework. We trained our_ft on all the five languages with the same training corpus, and kept the same hyperparameters as ft package except that we used 1024 as batch size for faster training. We show that our implementation yields comparable results with the original ft in word similarity and relatedness and parsing across languages. For entity typing, we experimented with the extended hierarchical composition architecture described in the paper on our_ft, i.e., using subword embeddings to compute word representations and updating subword embeddings of our_ft during training, and only use word embeddings as input for sgns and ft and directly update them. Additionally, we also experimented with some of our model configurations with character n-grams as our segmentation methods, but did not observe performance gains. Due to its large parameter space (usually 2-4 times larger than sgns) and the computational limits, we leave it as our future work.