To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
Table 3: Comparison of ELMO- cross-sentence embedding methods on dev. sets of sentence pair tasks.
['[EMPTY]', 'SICK-E', 'SICK-R', 'STS-B', 'MRPC']
Modelling pairwise interactions LSTMs consider each token sequentially, while Transformers can relate each token to every other in each layer Vaswani et al. This might facilitate with Transformers on sentence pair tasks, on which ELMo- performs comparatively poorly. To analyze this further, we compare different ways of encoding the sentence pair with ELMo and BERT. When adapting the ELMo LSTM to a sentence pair task, modeling the sentence interactions by fine-tuning through the bi-attention mechanism provides the best performance. This provides further evidence that the LSTM has difficulty modeling the pairwise interactions during sequential processing. This is in contrast to a Transformer LM that can be fine-tuned in this manner Radford et al.