Language Models with Transformers
Table 2: Ablation study. Compare CAS with not adding LSTM layers (CAS-Subset) and not updating Transformer block parameters (CAS-LSTM).
['[BOLD] Model', '[BOLD] Datasets  [BOLD] PTB', '[BOLD] Datasets  [BOLD] PTB', '[BOLD] Datasets  [BOLD] WT-2', '[BOLD] Datasets  [BOLD] WT-2', '[BOLD] Datasets  [BOLD] WT-103', '[BOLD] Datasets  [BOLD] WT-103']
but it fixes all Transformer blocks during fine-tuning. As can be seen, both CAS-Subset and CAS-LSTM improve significantly upon a naive use of BERT and GPT. This is to be expected since fine-tuning improves performance. On the smaller dataset, i.e. PTB, adding LSTMs is more effective. This might be due to the overfitting incurred in updating Transformers. On the other hand, on the larger datasets, i.e. WT-103, adding an LSTM is less effective, which means that adapting the Transformer parameters for better sentence-level representation is more important. Combing both together leads to further improvement. CAS outperforms AWD-LSTM-MoS on all three datasets.