Latent Multi-task Architecture Learning
Table 6: Ablation. Out-of-domain scores for Chunking with POS as auxiliary task, averaged across the 6 source domains.
['Task sharing', 'Layer sharing', 'bc', 'bn', 'mz', 'nw', 'pt', 'tc', 'wb', 'Avg']
Different types of sharing may be more important than others. We investigate the impact of i) the α parameters; ii) the β parameters; and iii) the division into subspaces with an orthogonality penalty. We also evaluate whether concatenation of the outputs of each layer is a reasonable alternative to our mixture model. Overall, we find that learnable α parameters are preferable over constant α parameters. Learned β parameters marginally outperform skip-connections in the hard parameter sharing setting, while skip-connections are competitive with learned β values in the learned α setting.