Large Scale Multi-Actor Generative Dialog Modeling
Table 4: Comparison of models with and without pre-trained initializations. All models in this study have 355M parameters with h=1024, l=24, and A=16.
['Model', 'P.T.', 'Iter.', 'Batch', 'PPL']
We observe that with less data, the pre-trained model quickly eclipses the model trained from scratch and achieves better perplexity, highlighting the need for models with robust linguistic features learned from non-Reddit corpora. Additionally, including reference history improves performance as well. This difference of 0.47, while smaller than differences between results from different model sizes, is notable due to the large amount of out of sample data that the models were tested on.