Context-Aware Self-Attention Networks
Table 1: Experimental results on WMT14 En⇒De translation task using Transformer-Base. “# Para.” denotes the trainable parameter size of each model (M = million). “Train” and “Decode’ denote the training speed (steps/second) and the decoding speed (sentences/second), respectively.
['[BOLD] #', '[BOLD] Model', '[BOLD] Applied to', '[BOLD] Context Vectors', '[BOLD] # Para.', '[BOLD] Train', '[BOLD] Decode', '[BOLD] BLEU']
Among them, global context (model #2) and deep context (model #4) gained comparable improvements. Deep-global context outperforms its global counterpart, showing that the different levels of global linguistic biases benefit to the accuracy of translation. Moreover, we evaluated whether the global and deep manners are complementary to each other. By simply summing them to the final context vectors, the model “deep-global context + deep context” (model #5) gains further improvements. According to the results, we argue that the two types of context vectors are able to improve the SANs in different aspects.