Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
Table 1: Experimental results on pairwise text classification and ranking tasks. Q-Att achieves comparable or competitive results compared with DeAtt with approximately one third of the parameter cost.
['Task Measure', 'NLI Accuracy', 'NLI Accuracy', 'NLI Accuracy', 'QA MAP/MRR', 'Paraphrase Accuracy', 'Paraphrase Accuracy', 'RS Top-1', '[EMPTY]']
We observe that a tiny Q-Att model (d=50) achieves comparable (or occasionally marginally better or worse) performance compared to DeAtt (d=200), gaining a 68% parameter savings. The results actually improve on certain data sets (2/7) and are comparable (often less than a percentage point difference) compared with the d=200 DeAtt model. Moreover, we scaled the parameter size of the DeAtt model to be similar to the Q-Att model and found that the performance degrades quite significantly (about 2%−3% lower on all data sets). This demonstrates the quality and benefit of learning with Quaternion space.