Globally Normalized Reader
Table 3: Impact of Beam Width B
['Model', '[ITALIC] B', 'EM', 'F1', 'Sentence']
In our first experiment, we aim to quantify the importance of global normalization on the learning and search process. We use T=104 Type Swap samples and vary beam width B between 1 and 32 for a locally and globally normalized models and summarize the Exact-Match and F1 score of the model’s predicted answer and ground truth computed using the evaluation scripts from (Rajpurkar et al., We additionally report another metric, the Sentence score, which is a measure for how often the predicted answer came from the correct sentence. This metric provides a measure for where mistakes are made during prediction. Thus, sentence selection is the easy part of the problem, and the model can allocate more computation (such as the end-word selection Bi-LSTM) to spans likely to contain the answer. This approach avoids wasteful work on unpromising spans and is important for further scaling these methods to long documents. Models such as Lee et al. Even with large beam sizes, the locally normalized model underperforms these approaches. However, by increasing model flexibility and performing search during training, the globally normalized model is able to recover from search errors and achieve much of the benefits of scoring all possible spans.