Comparative Analysis of Neural QA models on SQuAD
Table 2: Incorrect Answer Overlap (%)
['[BOLD] Model', '[BOLD] BiDAF', '[BOLD] R-Net', '[BOLD] DrQA', '[BOLD] DocQA', '[BOLD] LR']
Error Overlap Thus, the values in the table represent a symmetric matrix with diagonal elements indicating the number of errors which each model commits. This analysis can be useful while determining suitable models for creating meta ensembles since a low incorrect answer overlap indicates that the combined predictive power of the pair of models under consideration is high. We observe that most overlap values are in the range 20-25% indicating that an ensemble might give considerably better performance than individual models. DocQA paired with other models generates low values, as expected, but the least value is observed for the DocQA-DrQA pair probably because they both use very different feature representations and architectures, and hence generate diverse outputs. Note that DrQA is not the second best performing model (among the ones we analyzed) when considered independently, but might add more value to an ensemble because of the observed answer overlap trends.