DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding
Table 3: Ablation study results on Natural Questions Open.
['Retriever Model', 'Retriever P@10', 'Retriever Speedup', 'Answer EM']
Ablation study. To further investigate the impact of our model architecture design, we compare the performance of DC-BERT and its variants, including 1) DC-BERT-Linear, which uses linear layers instead of Transformers for interaction; and 2) DC-BERT-LSTM, which uses LSTM and bilinear layers for interactions following previous work Due to the simplistic architecture of the interaction layers, DC-BERT-Linear achieves the best speedup but has significant performance drop, while DC-BERT-LSTM achieves slightly worse performance and speedup than DC-BERT. When we increase the number of Transformer layers, the QA performance consistently improves, and the speedup decreases due to the increased computational cost. This shows the trade-off between the model capacity and efficiency of our method.