Unsupervised Text Generation byLearning from Search
Table 3: Model analysis on paraphrase generation. All variants use pretrained language models.
['Methods', 'iBLEU', 'BLEU', 'Inference Time']
Ablation study. We start from a base simulated annealing (SA) approach, where we have already adopted pretrained language models. Thus, it sets up a fair comparison. In particular, MM is better than CE by a significant margin in the second stage. Our intuition is that MM with negative samples in the beam makes tgls more sensitive in distinguishing sentence quality with its highly probable output region. Inference efficiency. The experiments were conducted on a cluster with Nvidia Telsa V100 GPUs. The inference time could be noisy due to the multi-thread nature of clusters, but it provides a conclusive enough comparison between search-based and autoregressive generation. As seen, SA is inefficient because it requires hundreds of steps of editing and reevaluation. SA+CE, SA+CE+SA+CE, and SA+CE+SA+CE are all based on the GPT2 model during inference, and thus are much more computationally efficient. Based on the validation, SA+CE adopts greedy decoding, whereas the other two adopt beam search with a size of 5. We see all GPT2-based generators are at least 6–10× faster than the search-based methods.