Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search
Table 3: Main results on the TREC Microblog 2011–2014 datasets. Rows are numbered in the first column, where each represents a model or a contrastive condition. The last row shows the relative improvement against QL. The best numbers on each dataset are in bold. Superscripts and subscripts indicate the row indexes for which a metric difference is statistically significant at p<0.05. Only methods 1–3 and 12–13 are compared with all other methods in the significance tests.
['[BOLD] ID', '[BOLD] Model  [BOLD] Metric', '[BOLD] 2011  [BOLD] MAP', '[BOLD] 2011  [BOLD] P30', '[BOLD] 2012  [BOLD] MAP', '[BOLD] 2012  [BOLD] P30', '[BOLD] 2013  [BOLD] MAP', '[BOLD] 2013  [BOLD] P30', '[BOLD] 2014  [BOLD] MAP', '[BOLD] 2014  [BOLD] P30']
Rows are numbered in the first column, where each represents a model or a contrastive condition. We compare our model to three sets of baselines: non-neural, neural, and interpolation. Interpolation methods are denoted by a symbol “+” at the end of the original model name, such as DUET+. Superscripts and subscripts indicate the row indexes for which a metric difference is statistically significant at p<0.05. However, RM3 requires an additional round of retrieval to select terms for query expansion, and thus is substantially slower. LambdaMART achieves effectiveness on par with RM3 when using all the hand-crafted features. From its contrastive variant with only text-based features, we can see that the overlap-based features provide little gain over QL. Comparing the rows “(text+URL)” and “(text+hashtag)” to row “(text)”, adding URL-based features leads to a significant improvement over text-based features, while hashtag-based features seem to bring fewer benefits. This confirms our observation (cf. that URLs appear frequently in tweets and contain meaningful relevance signals. Turning our attention to the last three rows, we observe that removing the character representations of URLs or documents both lead to significant drops across all datasets, with larger drops when URLs are removed. This suggests that URLs provide more relevance signals than character-level document modeling. Taking away the entire character-level module causes slightly more effectiveness loss. To conclude, the word-level matching module contributes the most effectiveness, but the character-level matching module still provides complementary and significantly useful signals.