Exploiting Sentential Context for Neural Machine Translation
Table 3: Performance on the linguistic probing tasks of evaluating linguistics embedded in the encoder outputs. “Base” denotes the representations from Tranformer-Based encoder. “Ssr” denotes shallow sentence representation. “Dsr” denotes deep sentence representation. “Avg” denotes the average accuracy of each category.
['[BOLD] Model', '[BOLD] Surface  [BOLD] SeLen', '[BOLD] Surface  [BOLD] WC', '[BOLD] Surface  [BOLD] Avg', '[BOLD] Syntactic  [BOLD] TrDep', '[BOLD] Syntactic  [BOLD] ToCo', '[BOLD] Syntactic  [BOLD] BShif', '[BOLD] Syntactic  [BOLD] Avg', '[BOLD] Semantic  [BOLD] Tense', '[BOLD] Semantic  [BOLD] SubN', '[BOLD] Semantic  [BOLD] ObjN', '[BOLD] Semantic  [BOLD] SoMo', '[BOLD] Semantic  [BOLD] CoIn', '[BOLD] Semantic  [BOLD] Avg']
From the tale, we can see that For different encoder layers in the baseline model (see “L4 in Base”, “L5 in Base” and “L6 in Base”), lower layers embed more about surface information while higher layers encode more semantics, which are consistent with previous findings in Raganato and Tiedemann Integrating the shallow sentence representation (“+ Ssr”) obtains improvement over the baseline on semantic tasks (75.33 vs. 74.61), while fails to improve on the surface (77.32 vs. 77.60) and syntactic tasks (64.88 vs. 65.00). This may indicate that the shallow representations that exploits only the top encoder layer (“L6 in Base”) encodes more semantic information. Introducing deep sentence representation (“+ Dsr”) brings more improvements. The reason is that our deep sentence representation is induced from the sentence representations of all the encoder layers, and lower layers that contain abound surface and syntactic information are exploited.