Reconstruction of Word Embeddings from Sub-Word Parameters
Table 2: Effect of reconstruction on word analogy (10 reconstruction epochs).
['Embedding', 'Syntactic', 'Semantic']
An interesting finding in our experiment is that for syntactic analogy, a randomly initialized character-based model outperforms the pre-trained embeddings and thus reconstruction only decreases the performance. We suspect that this is because much of the syntactic regularities is already captured by the architecture. Many questions involves only simplistic transformation, for instance adding r in wise: wiser∼free: x. The model correctly answers such questions simply by following its architecture, though it is unable to answer less regular questions (e.g., see:saw∼keep:x).