Source Dependency-Aware Transformer with Supervised Self-Attention
Table 1: Case-insensitive BLEU scores (%) for Chinese-to-English translation on NIST datasets. “+CSH” denotes model only trained under the supervision of child attentional adjacency matrix (β = 0). “+PSH” denotes model only trained under the supervision of parent attentional adjacency matrix (α = 0). “+CSH+PSH” is trained under the supervision of both.
['System', 'NIST2005', 'NIST2008', 'NIST2012', 'Average']
We report on case-insensitive BLEU here since English words are lowercased. From the table we can see that syntax-aware RNN models always outperform the RNNsearch baseline. However, the performance of the Transformer is much higher than that of all RNN-based methods. In Transformer+CSH, we use only the child attentional adjacency matrix to guide the encoder. Being aware of child dependencies, Transformer+CSH gains 1.0 BLEU point improvement over the Transformer baseline on the average. Transformer+PSH, in which only the parent attentional adjacency matrix is used as supervision, also achieves about 1.0 BLEU point improvement. After combination, the new Transformer+CSH+PSH can further improve BLEU by about 0.5 point on the average, which significantly outperforms the baseline and other source syntax-based methods in all test sets. This demonstrates that both child dependencies and parent dependencies benefit the Transformer model and their effects can be accumulated.