Controlled Text Generation for Data Augmentation in Intelligent Artificial Agents
Table 3: Best performance per metric for each model when applied to ‘all’ domains.
['training', 'training', 'S2S para', 'VAE recon', 'VAE para', 'VAE+DISC recon', 'CVAE recon', 'CVAE recon', 'CVAE para', 'CVAE+DISC recon']
For the intrinsic evaluation, we train the data generator either only on movie data or on ‘all’ data (movies and entertainment combined), using the respective dev sets for hyper-parameter tuning. During generation, we similarly consider either the movies test set, or the ‘all’ test set, and aim to generate ten synthetic phrases per test set phrase. VAE type generators can be trained for paraphrasing (c1→c2) or reconstruction (c1→c1). During generation, sampling can be performed either from the prior, e.g., by ignoring the encoder and sampling z∼N(0,I) to generate an output, or from the posterior e.g., using c1 as input to the encoder and producing the output c2. Note that not all combinations are applicable to all models. Special handling was required for a VAE with reconstruction training and prior sampling, where we have no control over the output signature. To solve this, we compared each output phrase to every signature in the train set (via BLEU4 Papineni et al. and assigned it to the highest scoring signature. Some sample output phrases can be seen in Fig. First, note the slot carry over (slot c.o.), which can be used as a sanity check measuring the chance of getting a phrase with the desired slot types. Most models reach 0.8 or higher slot c.o. as expected, but some fall short, indicating failure to produce the desired signature. The failure for VAE and CVAE models with discriminators is most notable, and can be explained by the fact that we have a large number of train signatures (∼800) and too few samples per signature (mean 8, median 4), to accurately train the discriminator. We verified that the discriminator overall accuracy does not exceed 0.35. The poor discriminator performance leads to the decoder not learning how to use signature s. The failure of VAE with posterior sampling is similarly explained by the large number of signatures: the signatures are so tightly packed in the latent space, that the variance of sampling z is likely to result in phrases from similar but different signatures.