Energy-Based Models for Text
Table 9: Validation and test perplexity on CC-News and Toronto Book Corpus. * denotes models initialized with RoBERTa trained on additional data. The joint model perplexity ranges are estimated using 100,000 samples, see Eq. 5. The number of parameters of each model is shown in parentheses.
['Model (#parameters)', 'CC-News Val', 'CC-News Test', 'Toronto Book Corpus Val', 'Toronto Book Corpus Test']
We can see that on both datasets, residual EBMs with causal attention joint UniT outperforms the baseline RALM with approximately the same number of parameters. The non-residual baseline BALM performs similarly to joint UniT, which might be due to the limitation that Pϕ is not trained jointly with the residual model in both joint UniT and RALM. However, by using our EBM approach, we can remove the causal attention mask and use bi-directional models, which achieves better performance than both baselines and joint UniT: without external data, joint BiT-Base reaches a higher performance than joint UniT with fewer parameters. By initializing from the state-of-the-art pretrained bi-directional transformers RoBERTa-Base and RoBERTa-Large, joint BiT-Base* and Joint BiT-Large* reach even better performance than joint BiT-Base.