State Gradients for RNN Memory Analysis
Table 2: Validation accuracy of logistic regression to predict POS-based classes. Numbers between brackets = number of target classes. N=nouns, V=verbs, Adj=adjectives, Adv=adverbs, Pro=pronouns.
['[BOLD] Class', '[BOLD] PTB-norm  [BOLD] cbow', '[BOLD] PTB-norm  [BOLD] LM emb', '[BOLD] WSJ-norm  [BOLD] cbow', '[BOLD] WSJ-norm  [BOLD] LM emb']
If we compare the type of word representations in the first two columns, we see that in most cases, the embeddings that are jointly trained with the LM (‘LM emb’) are better than the cbow embeddings in this task. Training the embeddings on more, in-domain, data (WSJ) also increases the classification accuracy. Using the embeddings of an LM trained on WSJ usually gives slightly better results than cbow embeddings, but given that the perplexity of the PTB LM with those embeddings is significantly higher (107.2 compared to 83.0 for cbow embeddings), we choose to focus on the cbow WSJ embeddings.