Synonyms and Antonyms: Embedded Conflict
Table 2: Comparison of two architectures on the FastText embeddings. For every architecture, a logistic regression and XGBoost classifiers were trained to distinguish two input vectors as synonyms of antonyms. Both architectures successfully transform the space of embeddings and highlight the differences between synonyms and antonyms. The baseline architecture that relies solely on the cosine similarity between obtained transformed embeddings performs better. Since Logistic Regression is based on the the assumption of linear separability of data its lower performance in comparison with XGBoost could be attributed to the nonlinearity of the encoded information.
['Architecture', 'Classifier', 'Raw', 'New', 'Raw + New']
The contrast map in this setup has the same amount of trainable parameters and produces the embeddings of the same dimension as the new embeddings produced by the baseline. However, a nonlinear trainable classifier contrasts the embeddings that are produced by the EmbeddingNet in this case. Therefore in the new embedding space the linear separability of the synonuyms and antonyms does not have to be improved. The Classification Net in the classifier system has a lot of trainable parameters, so the output of the EmbeddingNet does not have to highlight the differences between antonyms and synonyms so clearly since this difference could be further amplified by the Classification Net. However, the baseline does not have this luxury, and its output has to be as explicit as possible. Despite having comparable accuracy on the Raw Embeddings, XGBoost is superior on Transformed Embeddings, and the Transformed Embeddings concatenated with Raw Embeddings. This could be attributed to the fact that logistic regression is based on the the assumption of linear separability of data, which clearly does not hold for the embeddings in question.