Guiding attention in Sequence-to-sequence models for Dialogue Act prediction
Table 4: Accuracy on the dev set of the different encoder/decoder combination MRDA and SwDA. For SwDA, Wilcoxon test (10 runs with different seeds) has been performed for an HGRU encoder with a decoder with hard guided attention against an HGRU encoder with soft guided attention, soft guided attention, with attention, without attention pairwise tests exhibit p-value<0.01.
['[height=20pt]Enc.Dec.', 'SwDA VGRUD', 'SwDA att.', 'SwDA soft guid.', 'SwDA hard guid.']
For SwDA and MRDA, we observe that a seq2seq equipped with a hierarchical encoder outperforms models with Vanilla RNN encoder, while reducing the number of learned parameters. In NMT [attention], complementing a seq2seq with attention contributes to generate better sentences. In case of a seq2seq composed with a PersoHGRU and a decoder without attention the learning fails: the decrease of the training loss is relatively small and seq2seq fails to generalise. It appears that in DA classification where sequences are short (5 tags), Vanilla attention does not have as much as impact as in NMT (that have longer sequences with more complex global dependencies).