Improving Context Aware Language Models
Table 6: Results on Twitter data.
['[BOLD] Hidden ×', '[BOLD] Hidden +', '[BOLD] Output  [BOLD] LR', '[BOLD] Output  [BOLD] Hash', '[BOLD] PPL', '[BOLD] Acc.', '[BOLD] F1']
We compute both the perplexity and measure the performance of the models on a language identification task. In terms of perplexity, the best models do not make use of the multiplicative hidden layer adaptation, consistent with the results from the Reddit corpus. In general, the improvement in perplexity from adaptation is small (less than 5%) on this corpus compared to our other experiments where we saw relative improvements two to four times as big. This is likely because the LSTM can figure out by itself which language it is modeling early on in the sequence and adjust its predictions accordingly.