Stochastic Answer Networks for SQuAD 2.0
Table 2: Comparison with published results in literature. 1: results are extracted from Rajpurkar et al. (2018); 2: results are extracted from Hu et al. (2018). ∗: it is unclear which model is used. #: we only evaluate the Joint SAN in the submission.
['SQuAD 2.0  [BOLD] development dataset', 'SQuAD 2.0  [BOLD] development dataset EM', 'SQuAD 2.0  [BOLD] development dataset F1']
Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo). Comparing with the much complicated model R.M.-Reader + Verifier, which includes several components, our model still outperforms by 0.7 in terms of F1 score. Furthermore, we observe that ELMo gives a great boosting on the performance, e.g., 2.8 points in terms of F1 for DocQA. This encourages us to incorporate ELMo into our model in future.