Sparse Sequence-to-Sequence Models
Table 1: Average per-language accuracy on the test set (CoNLL–SIGMORPHON 2018 task 1) averaged or ensembled over three runs.
['[ITALIC] α output', '[ITALIC] α attention', 'high (avg.)', 'high (ens.)', 'medium (avg.)', 'medium (ens.)']
We report the official metric of the shared task, word accuracy averaged across languages. In addition to the average results of three individual model runs, we use an ensemble of those models, where we decode by averaging the raw probabilities at each time step. Our best sparse loss models beat the softmax baseline by nearly a full percentage point with ensembling, and up to two and a half points in the medium setting without ensembling. The choice of attention has a smaller impact. In contrast, our only departure from standard seq2seq training is the drop-in replacement of softmax by entmax.