Injecting Hierarchy with U-Net Transformers
Table 3: Ablation on the U-Net architecture. DOWN/UP represents the property that the number of tokens per layer decrease for each down layer, then increase for each up layer. CONV represents the addition of a convolutional layer. Note that without a convolutional layer, the number of tokens per layer remains constant. Conducted on the Cornell Movie Dialogues corpus.
['Architecture', 'Test Perplexity']
We start from the U-Net Transformer (bottom), and slowly modify it step by step into the vanilla Transformer, in order to analyze which changes resulted in performance gains. DOWN/UP represents the property that the number of tokens per layer decrease for each down layer, then increase for each up layer. CONV represents the addition of a convolutional layer. Note that without a convolutional layer, the number of tokens per layer remains constant.