AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses
Table 2: Automatic Evaluation results for the baselines and our proposed models (”iAUC” means ”inverted AUC”, or ”1 - AUC”; ”attn” means ”with attention”; ”s”, ”1”, ”2” and ”3” correspond to ”sentence-level”, ”unigram”, ”bigram” and ”trigram”, respectively; iAUC-avg is the average of all the other AUC columns). Best results are boldfaced. We do not calculate p-value because it does not apply to corpus-level metrics.
['[EMPTY]', '[ITALIC] iAUC-s', '[ITALIC] iAUC-1', '[ITALIC] iAUC-2', '[ITALIC] iAUC-3', '[ITALIC] iAUC-avg', 'Distinct-1', 'Distinct-2']
We can see that all our models have significantly better sentence-level diversity than VHRED, let alone LSTM. For unigram diversity, they are also better than LSTM, though hard to distinguish from VHRED. Both bigram and trigram graphs reveal that all models are more diverse than LSTM, except that RL shows lower diversity than the other models, which agree with our F1 results. Note that since our models are only trained based on unigram output distributions, the bigram and trigram diversities are still far away from that of the ground-truth, which points to future direction. That said, the table does show that encouraging unigram diversity can already have positive influence on higher grams as well. Also note that the hybrid model (last row) does not achieve the best result in terms of diversity. We hypothesize that this is because RL, which is usually harder to optimize than ML losses, faces exacerbated issues when combined with a strong MinAvgOut loss, which tries to pull the model output distribution away from the token distribution in the training corpus.