Enabling Language Models to Fill in the Blanks
Table 2: Human evaluation results. We use BERT (devlin2019bert), the best model from zhu2019text (SA), and our LM and ILM models to replace random sentences in five-sentence stories from the Stories test set. Then, we task humans with identifying which sentence of the five was generated by a machine. We report the score of each model: the percentage of infilled stories where the human failed to identify the machine-generated sentence. Our ILM model achieves a higher score than all of the other models. Note that the max score is effectively 80%, as a perfect model would cause annotators to randomly choose one of the five sentences.
['[EMPTY]', 'BERT', 'SA', 'LM', 'ILM']
For each approach we compute a score, which we define as the percentage of examples where the annotator did not correctly identify the machine-generated sentence. Therefore, a higher score implies a better (more natural, human-like) model.