Energy-Based Models for Text
Table 3: Number of parameters in millions for the discriminator. The computational cost is directly related to the number of parameters in other layers than the input embedding layer (second row).
['[EMPTY]', '[BOLD] Discriminators Linear', '[BOLD] Discriminators BiLSTM', '[BOLD] Discriminators BiLSTM Big', '[BOLD] Discriminators UniT', '[BOLD] Discriminators BiT']
We use data-parallel synchronous multi-GPU training with up to 24 nodes, each with 8 Nvidia V100 GPUs. The Wikitext dataset has lower accuracy because the discriminator overfits to such smaller dataset.