Improved Sentence Modeling using Suffix Bidirectional LSTM
Table 2: Performance of SuBiLSTM on the SentEval benchmark. The first 8 methods contain both unsupervised and supervised ones. FastSent is from [Hill, Cho, and Korhonen2016], SkipThought is described in [Kiros et al.2015], DisSent in [Nie, Bennett, and Goodman2018], CNN-LSTM in [Gan et al.2017], Byte mLSTM in [Radford, Józefowicz, and Sutskever2017], QuickThoughts in [Logeswaran and Lee2018] and MultiTask in [Subramanian et al.2018]. Our base model is InferSent [Conneau et al.2017]. Bold indicates the best performance among the SuBiLSTM models and the base model.
['[BOLD] Model', '[BOLD] MR', '[BOLD] CR', '[BOLD] SUBJ', '[BOLD] MPQA', '[BOLD] SST', '[BOLD] TREC', '[BOLD] MRPC', '[BOLD] SICK-R', '[BOLD] SICK-E', '[BOLD] STSB']
In all the tasks, SuBiLSTM and SuBiLSTM-Tied matches or exceeds the performance of the base model InferSent that uses a BiLSTM. For SuBiLSTM, among the classification tasks, the gains for SUBJ (0.8%), MPQA (0.5%) and TREC (1.6%) over InferSent are particularly notable. There is also a substantial gain of 1.2% in the semantic textual similarity task (STSB). The performance of SuBiLSTM-Tied also follows a similar trend, gaining 0.6% for SUBJ, 0.5% for SST , 2.2% for TREC and and 1.3% for STSB. The better performance on STSB is noteworthy as the sentence representations derived from a SuBiLSTM can take advantage of the long range dependencies it encodes. The 2-layer BiLSTM based model performs comparably to the single layer BiLSTM, despite using a much larger number of parameters.