Transformer on a Diet
Table 2: Results comparison (perplexity) of different Transformer language models on PTB and WT-2 data. Full: full Transformer; Dilated: dilated Transformer; Dilated-Memory: dilated Transformer with memory; Cascade: cascade Transformer.
['[BOLD] Model', '[BOLD] Parameter', '[BOLD] PTB  [BOLD] Val', '[BOLD] PTB  [BOLD] Test', '[BOLD] WT-2  [BOLD] Val', '[BOLD] WT-2  [BOLD] Test']
We compare the effectiveness of the proposed Transformer architectures with the full-Transformer architecture. Transformer performs closely to the full-Transformer structure. This indicates that local context is very important to language model tasks, and cascade Transformer is able to capture the meaning local dependency. We also compare the parameter sizes between light Transformers and full Transformer. Among all the architectures, dilated Transformer is lightest one. Although it delivers moderate results, however, when compare to full Transformer, we save 70% model size and the computation could be more efficient. It would suggest the best Transformer architecture given the deployment constraints, such as model size limit, latency requirement or the quality.