Semi-supervised Question Retrieval with Gated Convolutions
Table 2: Comparative results of all methods on the question similarity task. Higher numbers are better. For neural network models, we show the best average performance across 5 independent runs and the corresponding pooling strategy. Statistical significance with p<0.05 against other types of model is marked with ∗.
['Method', 'Pooling', 'Dev MAP', 'Dev MRR', 'Dev P@1', 'Dev P@5', 'Test MAP', 'Test MRR', 'Test P@1', 'Test P@5']
The results show that our full model, RCNNs with pre-training, achieves the best performance across all metrics on both the dev and test sets. For instance, the full model gets a P@1 of 62.0% on the test set, outperforming the word matching-based method BM25 by over 8 percent points. Further, our RCNN model also outperforms the other neural encoder models and the baselines across all metrics. This superior performance indicates that the use of non-consecutive filters and a varying decay is effective in improving traditional neural network models. The RCNN model when pre-trained on the entire corpus consistently gets better results across all the metrics. Interestingly, the LSTM and GRU networks obtain similar perplexity on the heldout set, but achieve much worse MRR for similar question retrieval. For instance, the GRU encoder obtains only 63% MRR, 5% worse than the RCNN model’s MRR performance.