Large Scale Multi-Actor Generative Dialog Modeling
Table 3: Comparison of model architecture perplexity (PPL) trained from scratch for 200K iterations. The top half of the table are iso-parameter count experiments while the bottom half are iso-architecture experiments. Note that h, l, and A define the sizes of the encoder and decoder transformers individually.
['Model', '[ITALIC] h', '[ITALIC] l', '[ITALIC] A', 'Params', 'PPL']
We see that for models with similar parameter counts the GCC-Dec has the advantage, and that under similar decoder sizes having direct access to the reference material (i.e. processing the reference and conversation together in a single decoder) results in superior performance. This indicates that the added complexity from additional encoding is not needed and that concatenating all relevant context is both the simplest, and most effective means of incorporating previous information. Since the parameters are shared and no latent variable bottleneck is used, the model has full access to the information from the references. With this, the self attention operation is able to automatically modify the model’s output distribution in complex, non-linear ways without the need for manual architecture design.