A Novel Neural Sequence Model with Multiple Attentions for Word Sense Disambiguation
TABLE IV: Change in different attention weights with epochs for Seq2Seq + conv + POS (weighting).
['[BOLD] Attention weights', '[BOLD] Initial value', '[BOLD] 1000th epoch', '[BOLD] 2000th epoch', '[BOLD] 4000th epoch', '[BOLD] 5000th epoch', '[BOLD] 3200th epoch']
It can be easily seen that, with more data the model sees, it gives more more importance to POS with weight 3.58 compared to the other two (0.32 for words and 3.45 for bigrams). The weight on word attention is not stable but weights on the other two are increasing monotonically until the loss gets very small. Finally, we select the set of weights with which the model achieves highest F1 score at the 3200th epoch.