Encoder-Agnostic Adaptation forConditional Language Generation
Table 3: Story generation on the WritingPrompts dataset. Rank acc. refers to the top-1 prompt ranking accuracy metric described in Section 4.3. (Experiments use the GPT2 BPE scheme, so PPL numbers are not directly comparable to those reported in (Fan et al., 2018)). Bold indicates statistically significant best results at p≤0.05.
['Model', 'CIDEr ↑', 'B4 ↑']
Despite the large dataset size, the Repr-Transfomer and Pseudo-Self approaches still substantially reduce the PPL. That the models are able to improve PPL, despite the 100 million+ target tokens, suggests these models are able to effectively make use of the GPT-2 LM. Pseudo-Self sees only a 0.3% decrease in prompt ranking accuracy, while the Repr-Transformer approach sees a larger decrease. The Context-Attn model runs into optimization challenges and fails to learn in this setting. We compare the three transfer learning methods with a non-pretraining baseline and models from the literature. Of the three pretraining approaches Pseudo-Self gives the best performance, and is the only model to improve both CIDEr and BLEU-4 compared to the Transformer baseline. Furthermore, Pseudo-Self outperforms all other models on CIDEr but gives a slightly worse BLEU-4.