Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
Table 10: Development and test accuracy of source tasks (Quora, SNLI and Multi-NLI) obtained through various multi-task learning architectures. Underlined values indicate best performance among models trained on same set of tasks and bold-faced values indicate best performance among all models.
['Model Type', 'Quora dev', 'Quora test', 'SNLI dev', 'SNLI test', 'Multi-NLI dev', 'Multi-NLI test']
In this section, we discuss the performance of both fully shared and shared-private multi-task learning (MTL) frameworks on different combinations of DupQuD, SNLI and Multi-NLI datasets as source tasks. For the shared-private models, we concatenate the representations generated by shared and private (task-specific) encoders to form sentence embeddings. We compare the performance of MTL with the models trained on single tasks as in Conneau et al. This is intuitive because when more tasks are involved, it becomes harder to get good performance by modeling only the commonalities. So, having both task-specific and common features and allowing the final classifier to combine them is a sensible choice. However, to our surprise, the adversarial training does not always excel on source tasks but in the next section, we will show that adversarial training boosts the transfer learning performance.