Ensembling Strategies for Answering Natural Questions
Table 3: Impact of candidate pool size. All experiments run with a greedy search strategy optimized exclusively for long answer F1 without normalization using the max aggregator.
['[ITALIC] NS', '[ITALIC] SA\xa0F1 [ITALIC] Tr', '[ITALIC] LA\xa0F1 [ITALIC] Tr', '[ITALIC] SA\xa0F1 [ITALIC] Te', '[ITALIC] LA\xa0F1 [ITALIC] Te']
We use the greedy search strategy to optimize for long answer performance without normalization using the max aggregator. For the experiment with 20 candidates we use the same 20 candidates as for the exhaustive search strategy experiment (greatest sum of short and long answer F1s). We find that the added diversity of the models avaible in the entire pool produces a stronger ensemble which generalizes better both in short and long answer F1. Having verified the benefit of using the entire candidate pool we continue to do so in the remainder of our experiments. The max experiment is as-before run without normalization to greedily optimize for long answer F1. The other experiments here are normalization with logistic regression, as our experiments showed that not normalizing decreased performance. We find that using max aggregation results in the best short answer F1 whereas using normalized noisy or aggregation results in the best long answer F1. Based on these results, we run a final experiment using unnormalized max for short answers and logistic regression normalized noisy or works for long answers. We find that this approach produces the strongest performance for both short and long answers with 0.5934 SA F1Te and 0.7150 LA F1Te. These numbers translate to a full dev performance of 0.5933 short answer F1 and 0.7107 long answer F1, which represents an improvement of 2.3 short answer F1 and 4.0 long answer F1 over our best single model.