Deeper Text Understanding for IR with Contextual Neural Language Modeling
Table 3. Accuracy on different types of Robust04 queries. Percentages show relative gain/loss over title queries.
['Query', 'Avg Len', 'nDCG@20 SDM', 'nDCG@20 SDM', 'nDCG@20 Coor-Ascent', 'nDCG@20 Coor-Ascent', 'nDCG@20 BERT-MaxP', 'nDCG@20 BERT-MaxP']
This section examines BERT on 3 types of queries that require different levels of text understanding: title, description, and narrative. To test the effect of grammar structures, a keyword version of description and narrative is generated by removing stopwords and punctuation. To test how BERT understands the logic in narratives, a ‘positive’ version of narrative is generated by removing negative conditions (e.g. ”Not relevant are documents…”). Supervised methods use narratives to re-rank title query initial results due to low recall of BOW on narratives, which gives narratives an advantage over the other types of queries. SDM works best with titles. Coor-Ascent is moderately better with descriptions and narratives. The two methods weight words solely based on term frequencies, but word importance should depend on the meaning of the entire query. In contrast, BERT-MaxP makes large improvement on longer queries by modeling word meaning and context. Keywords versions perform better than the original query for SDM and Coor-Ascent, because stopwords are noisy to traditional match signals like TF. In contrast, BERT is more effective on the original natural languages queries. Although stopwords and punctuation do not define the information need, they build the structures in a language. BERT is able to capture such structures, achieving a deeper query understanding than flat bag-of-words. It is unable to leverage evidence from negative logic conditions in narratives; removing negative conditions does not hurt performance.