Frozen Binomials on the Web: Word Ordering and Language Conventions in Online Text
Table 9. Results of logistic regression based on word embeddings. This is by far our most successful model. Note that not all words in our binomials were found in the word embeddings, leaving about 70–97% of the binomials usable.
['[EMPTY]', 'Accuracy']
Word Embeddings. Word embeddings assign a vector xi to each word i in the corpus, such that the relative position of these vectors in space encode information lingustically relevant relationships among the words. Using the Google News word embeddings, via a simple logistic model, we produce a vector v∗ and predict the ordering of a binomial on words i and j from v∗⋅(xi−xj). In this sense, v∗ can be thought of as a “sweep-line” direction through the space containing the word vectors, such that the ordering along this sweep-line is the predicted ordering of all binomials in the corpus. This is by far the best prediction method we tested. It is important to note that not all words in our binomials could be associated with an embedding, so it was necessary to remove binomials containing words such as names or slang. However, retesting our basic features on this data set did not show any improvement, implying that the drastic change in predictive power is not due to the changed data set.