KERMIT: Generative Insertion-Based Modeling for Sequences
Table 4: SQuAD zero-shot cloze question answering.
['[BOLD] Model', '[BOLD] Exact Match', '[BOLD] F1']
We evaluate on 50,000 cloze-formulated questions from SQuAD, using the standard SQuAD evaluation script to compute accuracy in terms of exact match and token-level F1. KERMIT performs significantly better on this zero-shot cloze task than the other two approaches thanks to its infilling capabilities learned through its insertion-oriented objective, achieving 30.3 F1 and 20.9% exact match. BERT ’s performance falls short of KERMIT, as it often prefers shorter completions since it is not required to handle length modeling during training. GPT-2 lags further behind the others due to its inability to condition on the context on both sides of the gap during inference. Even when the oracle length (i.e., the ground-truth length of the answer) is provided to BERT and GPT-2, KERMIT still substantially outperforms all other models.