Enabling Language Models to Fill in the Blanks
Table 3: Document infilling PPL (or language modeling) of ILM and baselines initialized either from scratch or from the pre-trained checkpoint across three datasets. Note that PPL of ILM is similar to LM, implying that our infilling strategy can reasonably maintain the ability to perform language modeling while extending the ability to infill.
['[EMPTY]', 'Sto', 'Abs', 'Lyr']
Because of how we structured our mask function We see that performance of ILM is similar to that of LM on this task, even though ILM sees far fewer examples of language modeling compared to LM.