Hierarchical Character-Word Models for Language Identification
Table 5: Top seven most similar words from the training data and their cosine similarities for inputs “couldn’t”, “@maria_sanchez”, and “noite”.
['[BOLD] couldn’t can’t', '[BOLD] couldn’t 0.84', '[BOLD] @maria_sanchez @Ainhooa_Sanchez', '[BOLD] @maria_sanchez 0.85', '[BOLD] noite Noite', '[BOLD] noite 0.99']
In the left column we see that words with similar features, such as the presence of “n’t” contraction, can be grouped together by char2vec. In the middle column, an out-of-vocabulary username is supplied and similar usernames are retrieved. When working with n-gram features, removing usernames is common, but some previous work demonstrates that they still carry useful information for predicting the language of the tweet The third example,“noite” (the Portuguese word for “night”), shows that the word embeddings are largely invariant to changes in punctuation and capitalization.