TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
Table 1: Parameter size and training speed for TRANS/BERT, TRANS-BLSTM-SMALL, and TRANS-BLSTM on base and large settings respectively.
['[EMPTY]', 'Model', 'Parameters (M)', 'Layers', 'Hidden', 'Embedding', 'Heads', 'Speedup']
Here, the TRANS-BLSTM-SMALL and TRANS-BLSTM models are 50% and 100% larger than the TRANS model (base, large) respectively. Consequently, TRANS-BLSTM-SMALL and TRANS-BLSTM models require more computational resources and longer training times compared to the vanilla transformer model. The slowest-training model is the TRANS-BLSTM which is also our baseline. Models with fewer parameters can train faster. For example, the large TRANS/BERT model boasts a 2.8 fold speedup compared to the TRANS-BLSTM large model. We note that the focus of the paper is to investigate whether a joint transformer and BLSTM architecture can further improve the performance over a transformer baseline. This is important to keep in mind because simply increasing the number of hidden units in BERT-large is not enough to positively affect accuracy Lan et al.