Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
Table 6: Comparisons of various neural baselines on the TREC QA task on two dataset settings TRAIN and TRAIN-ALL. Competitors (except QRNN and CTRN) are reported from [Tay et al.2017]. Best result is in boldface and second best is underlined.
['[BOLD] Model', '[BOLD] TRAIN  [BOLD] MAP', '[BOLD] TRAIN  [BOLD] MRR', '[BOLD] TRAIN-ALL  [BOLD] MAP', '[BOLD] TRAIN-ALL  [BOLD] MRR']
CTRN achieves the top results comparing to the multitude of neural baselines. Notably, the performance of CTRN is about 1%−5% better than QRNN. QRNN performs very competitively on the TRAIN-ALL setting but fails in comparison for the TRAIN setting. This might be because QRNN, with three 1D convolutional layers, might overfit on the smaller dataset. However, CTRN performs well on smaller TRAIN as well which hints at possibly some regularizing effect of the LTC mechanism. The performance of the vanilla QRNN model on the TRAIN-ALL setting is also surprisingly competitive, outperforming more complex models such as HD-LSTM and NTN-LSTM.