Blank Language Models
Table 1: Perplexity on the Penn Treebank and WikiText datasets.
['[EMPTY]', 'PTB', 'WT2', 'WT103']
The top results are achieved by the Transformer-XL However, these systems have additional advantages in terms of utilization of supplementary techniques and their model size. On WikiText-103, the models in Dai et al. These advancements can also be combined with our model. When evaluated against comparable baselines, our model rivals the Insertion Transformer as well as left-to-right language models with LSTM and Temporal Convolutional Network (TCN) architecture. The finding is particularly noteworthy, since the language modeling task is more challenging for free-order models like ours.