RikiNet: Reading Wikipedia Pages for Natural Question Answering
Table 3: Ablations of multi-level cascaded predictor on dev set of NQ dataset.
['Setting', 'LA F1', 'SA F1']
On the predictor side, we further remove or replace its component and report the results in Tab. In (1) we remove the whole DPDA reader and predictor. In (2), we remove the way of multi-level prediction (i.e., training the model to predict long and short answer jointly) We can see that our multi-level prediction is critical to the long answer prediction. In (3) we only remove the cascaded structure but keep the multi-level prediction, which means that the prediction representations are no longer used as input for other predictions, the performance of both long and short answers drops about 1.0 F1 score. In (4) we change the ordering of cascaded process. , we consider the cascaded structure of short answer first and then long answer. However, we get slightly worse results in this way. In (5), we remove the question embedding which is used for answer type prediction. It can be observed that the question embedding contributes to performance improvement. In the variants of (6)-(9), we remove the dense prediction layers with Tanh activation function and replace it with Bi-directional Long-Short Term Memory (Bi-LSTM) hochreiter1997long; schuster1997bidirectional layers, transformer self-attention blocks, and dense prediction layers with Gaussian Error Linear Unit GELU hendrycks2016gaussian activation function but neither get better performance.