Learning to Encode Position for Transformer with Continuous Dynamical Model
Table 5: Experiment results on SQuAD benchmark. All results are obtained from RoBERTa-large model.
['Model', '[BOLD] SQuAD 1.1 EM', '[BOLD] SQuAD 1.1 F1', '[BOLD] SQuAD 2.0 EM', '[BOLD] SQuAD 2.0 F1']
In this dataset, each item contains a lengthy paragraph containing facts and several questions related to the paragraph. The model needs to predict the range of characters that answer the questions. In SQuAD-v2, the problem becomes more challenging that the questions might be unanswerable by the context. We follow the same data processing script as BERT/RoBERTa for fair comparison; more details about the training process are described in the Supplementary material. As we can see, the FLOATER model beats the baseline RoBERTa model consistently across most datasets. The improvement is significant, considering that both models are finetuned from the same pretrained checkpoint.