BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
Table 2: Mean precision at one (P@1) for BERT-base, BERT-large, the k-NN search and the interpolation between BERT and the k-NN search (BERT-kNN) across the set of evaluation corpora.
['[BOLD] Corpus', '[BOLD] Relation', '[BOLD] Statistics Facts', '[BOLD] Statistics Rel', '[BOLD] model BERT-base22footnotemark: 2', '[BOLD] model BERT-large††footnotemark:', '[BOLD] model kNN', '[BOLD] model BERT-kNN']
We see that BERT-kNN outperforms BERT-base and BERT-large on 3 out of 4 LAMA subsets. On ConceptNet it shows competitive results. Huge gains are obtained on the GoogleRE dataset. BERT-kNN performs better in all three categories. Only the interpolation of the two yields optimal results. In many cases, the knowledge recalled by BERT and the kNN is complementary. BERT is much better on ConceptNet relations. This seems to be due to the limitations of knowledge expressed in Wikipedia articles. Note that the interpolation parameter is kept constant over all datasets. The prediction probabilities are well calibrated in a sense that BERT-kNN is able to distinguish when to rely more on BERT or the kNN predictions.