Tree Transformer: Integrating Tree Structures into Self-Attention
Table 4: The perplexity of masked words. Params is the number of parameters. We denote the number of layers as L. In Transformer L=10−B, the increased hidden size results in more parameters.
['[BOLD] Model', '[BOLD] L', '[BOLD] Params', '[BOLD] Perplexity']
All hyperparameters except the number of layers in Tree Transformer and Transformer are set to be the same and optimized by the same optimizer. We use adam as our optimizer with learning rate of 0.0001, β1=0.9 and β2=0.999. Our proposed Constituent Attention module increases about 10% hyperparameters to the original Transformer encoder and the computational speed is 1.2 times slower. The results with best performance on validation set are reported. Compared to the original Transformer, Tree Transformer achieves better performance on masked language modeling. As the performance gain is possibly due to more parameters, we adjust the number of layers or increase the number of hidden layers in Transformer L=10−B. Even with fewer parameters than Transformer, Tree Transformer still performs better.