Low-Rank RNN Adaptation for Context-Aware Language Modeling
Table 5: The top boosted words in the Softmax bias layer for different context settings in a FactorCell model.
['[BOLD] Hotel', '[BOLD] City', '[BOLD] Class', '[BOLD] Rating', '[BOLD] Top Boosted Words']
Another way of understanding what the context embeddings represent is to compute the softmax bias projection Qc and examine the words that experience the biggest increase in probability. In each case, the top words are strongly related to geography and include names of neighborhoods, local attractions, and other hotels in the same city. The top boosted words are relatively unaffected by changing the rating. (Recall that the hotel identifier and the user rating are the only two inputs used to create the context embedding.) This table combined with the other visualizations indicates that location effects tend to dominate in the output layer, which may explain why the two models adapting the recurrent network seem to have a bigger impact on classification performance.