Same Representation, Different Attentions:A New Scheme of Information Sharing for Multi-task Learning
Table 2: Performances on 16 tasks. The column of “Single Task” includes bidirectional LSTM (BiLSTM), bidirectional LSTM with attention (att-BiLSTM) and the average accuracy of the two models. The column of “Multiple Tasks” shows several multi-task models. * is from [Liu et al.2017] .
['[BOLD] Task', 'Single Task BiLSTM', 'Single Task att-BiLSTM', 'Single Task Avg.', 'Multiple Tasks FS-MTL', 'Multiple Tasks SSP-MTL', 'Multiple Tasks PSP-MTL', 'Multiple Tasks ASP-MTL*', 'Multiple Tasks SA-MTL', 'Multiple Tasks DA-MTL']
From the table, we can see that the performances of most tasks can be improved with the help of multi-task learning. FS-MTL shows the minimum performance gain from multi-task learning since it puts all private and shared information into a unified space. SSP-MTL and PSP-MTL achieve similar performance and are outperformed by ASP-MTL which can better separate the task-specific and task-invariant features by using adversarial training. Our proposed models (SA-MTL and DA-MTL) outperform ASP-MTL because we model a richer representation from these 16 tasks. Compared to SA-MTL, DA-MTL achieves a further improvement of +0.6 accuracy with the help of the dynamic and flexible query vector. It is noteworthy that our models are also space efficient since the task-specific information is extracted by using only a query vector, instead of a BiLSTM layer in the shared-private models.