TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation
Table 1: Results for all our methods and the baselines. ‘w/ pre-attn’ and ‘w/ uniform’ are short for ‘with pre-attention’ and ‘with weight of 2 uniformly’. Best scores in each column within each row-wise block are highlighted in bold face. ↑/↓ indicate higher/lower is better. All results are on the test set.
['[EMPTY]', '[EMPTY]', 'Dailydialog WL2↓', 'Dailydialog BLEU↑', 'Dailydialog l-DIMEN↑', 'Movie Dialogs WL2↓', 'Movie Dialogs BLEU↑', 'Movie Dialogs l-DIMEN↑', 'Ubuntu Dialogue WL2↓', 'Ubuntu Dialogue BLEU↑', 'Ubuntu Dialogue l-DIMEN↑']
Since the validation repetition (WL2 scores) of models can be very low in the early stage of training when the l-DIMEN score is still very low, we mainly use l-DIMEN to make sure that the results are shown for checkpoints at similar stages. The first block ranging from rows (a)–(c) is for answering RQ1. The second (rows (d)–(g)) and third (rows (h)–(l)) blocks are for answering RQ2 and RQ3, with RNNs and transformer models, respectively. Answer to RQ1: From row (b), we can see that pre-attention is effective in reducing repetition on all three datasets. On top of pre-attention, row (c) shows that highway connections are helpful with repetition reduction on two datasets, namely Dailydialog and Movie Dialogs. BLEU scores show that the quality of the utterances generated by our models (b) and (c) are on a par with those generated by the baseline (a). These results indicate that proper architectural modifications can indeed reduce repetition, and more work needs to be done to find such modifications for transformers. Answer to RQ3: Next, we proceed to our proposed weighting methods. Though TLDR occasionally performs worse than TFL, e.g., row (g) vs row (e) on the Movie Dialogs dataset, we note that this is probably due to sample bias of the test sets. These comparisons indicate that dynamically weighting tokens according to their difficulty is an effective solution to reducing repetition.