Exploring sentence informativeness
Table 4: The table on the left shows the mean similarity (for the 20 target words) between the pre-trained vector of the target words and the vector trained from selecting only 250 sentences with highest and lowest probability as well as 250 random sentences according to the Fasttext classifier’s output. The second table shows the effect that adding 50 low probability sentences to 200 random sentences has on the mean similarity.
['[EMPTY]', '[BOLD] sim-inf', '[BOLD] sim-uninf', '[BOLD] sim-inf&uninf', '[BOLD] sim-rand250', '[BOLD] sim-rand200', '[BOLD] sim-rand&uninf']
sim-inf, sim-uninf and sim-random are the similarities computed on the sets of 250 most informative sentences, 250 least informative and 250 random, respectively; sim-inf&uninf includes the 250 most informative sentences and 250 least informative. Looking at the mean differences of each column with sim-inf, we conclude that a low informativeness score of a sentence towards a target word indicates it is less suitable to learn a word embedding; however, sentences with high probability are not necessarily more helpful than the rest. Moreover, training embeddings on a set of 500 good and bad sentences gives almost the same quality of embeddings as training on a set of only 250 good sentences. The former is usually lower when the sim-uninf value is low, showing that removing these sentences can improve word representations. However, for words with a high value of sim-uninf, the value sim-random-uninf is still higher than sim-random200. Thus, the mean difference is close to zero. We investigate the reason behind the large disparities among words for this task. We consider two indicators for each word: its frequency in Google Unigrams brants2006web and its polysemy in WordNet Fellbaum1998. The Spearman correlation between the word frequency and the difference between sim-inf and sim-random is high and negative (-0.6) with a p-value <0.05. Thus, for frequent words, the difference in similarity is low between highly informative sentences and random sentences. On the contrary, in the case of infrequent words, the informative sentences provide better embeddings than the random ones. For polysemy, we divide the words into two classes according to the median: the words with less than 11 different senses in WordNet and the words with 11 or more. No significant correlation is observed with the similarity values.