Adaptability of Neural Networks on Varying Granularity IR Tasks
Table 2: Comparison of different models over the Robust04 collection with title queries. * means significant difference over QL respectively at 0.005 significance level measured by Fisher randomization test.
['Method', 'MAP', 'nDCG@20', 'P@20']
Varied input length. In ad-hoc retrieval tasks, the length differences in documents are so large that they significantly affect the training of deep models. Most neural models include a step that converts varied length input into fixed length vectors (i.e. input layer for DNN, pooling in CNN and memory vectors in RNN). Without accounting for the original length of text, this process could introduce strong bias for short or long documents. For example, to train a deep structure semantic model (DSSM) for ad-hoc retrieval, Huang et al. When applied to short text like web page titles, which have few n-grams, word hashing produces high quality representations without losing too much information. However, this technique becomes problematic as document length increases from tens of words to hundreds of words. According to our observations, the n-gram representations for documents with hundreds of words are dense and noisy. Unsurprisingly, our experiments with DSSM on standard ad-hoc retrieval collections were not effective. The MAP of DSSM and its convolution version (CLSM) are less than 0.1 on Robust04 title queries. WE Using WE and PV solely did not perform well compared to QL with dirichlet smoothing. We only achieve positive results when we combined these models with language modeling approaches that explicitly capture the exact matching information of queries and documents. As these problems pose significant challenges from a deep learning perspective, one direction of future research is examining the role of the attention mechanism when dealing with documents at the coarse granularity level.