Derivational Morphological Relations in Word Embeddings
Table 4: Results on vectors learned by the NMT models compared to word2vec. K-means clustering with 21 clusters. The results are averaged over 10 independent runs.
['model', 'cls', 'H', 'C', 'V', 'A']
The cls columns shows how many different classes are assigned. Because some classes are more frequent than others, they may form the majority in multiple clusters. This is why random baseline assignes less than 3 different classes on average. We see that word2vec (trained on the Czech side of the parallel corpus) captures more information about derivations than NMT models. RNN models store more information in the embeddings if they do not utilize the attention mechanism. Even less information is stored in the embeddings by the Transformer architecture. This is probably because while in attention-less model the embedding is the only set of parameters directly associated with the given word, in the attention model the information can be split between embeddings and the attention weights. The transformer architecture with residual connections has even more parameters associated with a given word. Decoder in general stores more information about relation between words in the embeddings than encoder, presumably because it partially supplies the role of a language model. This clustering has a high V-measure, because its completeness is 100 % (the super+ class is not present in the NMT data and for all the other classes it holds that each member of a class has the same parent-child POS tags pair). But it has lower accuracy than all the other models (except for the random baseline), showing that the unsupervised clustering does more than just clustering by POS.