Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
Table 3: Experimental results on neural machine translation (NMT). Results of Transformer Base on EN-VI (IWSLT 2015), EN-RO (WMT 2016) and EN-ET (WMT 2018). Parameter size excludes word embeddings. Our proposed Quaternion Transformer achieves comparable or higher performance with only 67.9% parameter costs of the base Transformer model.
['Model', 'BLEU IWSLT’15  [ITALIC] En-Vi', 'BLEU WMT’16  [ITALIC] En-Ro', 'BLEU WMT’18  [ITALIC] En-Et', '# Params']
On the IWSLT’15 En-Vi data set, the partial adaptation of the Quaternion Transformer outperforms (+2.5%) the base Transformer with a 32% reduction in parameter cost. On the other hand, the full adaptation comes close (−0.4%) with a 75% reduction in paramter cost. On the WMT’16 En-Ro data set, Quaternion Transformers do not outperform the base Transformer. We observe a −0.1% degrade in performance on the partial adaptation and −4.3% degrade on the full adaptation of the Quaternion Transformer. However, we note that the drop in performance with respect to parameter savings is still quite decent, e.g., saving 32% parameters for a drop of only 0.1 BLEU points. The full adaptation loses out comparatively. On the WMT’18 En-Et dataset, the partial adaptation achieves the best result with 32% less parameters. The full adaptation, comparatively, only loses by 1.0 BLEU score from the original Transformer yet saving 75% parameters.