Neuron Interaction Based Representation Composition for Neural Machine Translation
Table 3: Classification accuracies on 10 probing tasks of evaluating the linguistic properties (“Surface”, “Syntactic”, and “Semantic”). “Ave.” denotes the averaged accuracy in each category. “△” denotes the relative improvement, and we highlight the numbers ≥1%.
['[BOLD] Task  [BOLD] Surface', '[BOLD] Task SeLen', '[BOLD] Base 92.20', 'Ours 92.11', '△ -0.1%']
The NI-based composition outperforms that by the baseline in most probing tasks, proving that our composition strategy indeed helps Transformer encoder generate more informative representation, especially at the syntactic and semantic level. The averaged gains in syntactic and semantic tasks are significant, showing that our strategy makes San capture more high-level linguistic properties. Note that the lower values in surface tasks (e.g., SeLen), are consistent with the conclusion in [conneau2018acl]: as model captures deeper linguistic properties, it will tend to forget about these superficial features.