Learning to Search in Long Documents Using Document Structure
Table 5: Navigation accuracy for all models on the development set of TriviaQA-NoP.
['[EMPTY]', 'Development EM', 'Development F1', 'Test EM', 'Test F1']
Focusing on navigation accuracy, we see that randomly walking or choosing a paragraph yields low performance. Vanilla Tf-Idf performs considerably better than the random baselines, but is outperformed by all other models. Comparing DQN to DocQN, we see that DocQN outperforms DQN. Allowing DQN and DocQN to run the RC model during navigation improves their performance, but the accuracy gain is substantially higher for DocQN (2.3% accuracy and 3.1% aggregated accuracy) than for DQN (0.4% accuracy and 0.7% aggregated accuracy). Doc-Tf-Idf, which has access to the entire document outperforms DocQN, which consumes on average 6% of the entire document. Nonetheless, the two models obtain the same aggregated accuracy. This good performance of Doc-Tf-Idf shows that in TriviaQA-NoP answer paragraphs share a lot of lexical material with the question. Importantly, an ensemble of Doc-Tf-Idf with DocQN substantially improves the overall performance, reaching accuracy of 35% and aggregated accuracy of 48%. This is in sharp contrast to an ensemble with DQN, where for any value of l, Doc-Tf-Idf performs better on its own.