Document-Level Neural Machine Translation with Hierarchical Attention Networks
Table 4: Evaluation on discourse phenomena. Noun and pronoun translation: Accuracy with respect to a human reference. Lexical cohesion: Ratio of repeated and lexically similar words over the number of content words. Coherence: Average cosine similarity of consecutive sentences (i.e. average of LSA word-vectors)
['[EMPTY]', '[BOLD] Noun Translation  [BOLD] TED Talks', '[BOLD] Noun Translation  [BOLD] TED Talks', '[BOLD] Noun Translation  [BOLD] Subtitles', '[BOLD] Noun Translation  [BOLD] Subtitles', '[BOLD] Noun Translation  [BOLD] News', '[BOLD] Pronoun Translation  [BOLD] TED Talks', '[BOLD] Pronoun Translation  [BOLD] TED Talks', '[BOLD] Pronoun Translation  [BOLD] Subtitles', '[BOLD] Pronoun Translation  [BOLD] Subtitles', '[BOLD] Pronoun Translation  [BOLD] News']
For nouns, the joint HAN achieves the best accuracy with a significant improvement compared to other models, showing that target and source contextual information are complementary. Similarity for pronouns, the joint model has the best result for TED talks and news. However, HAN encoder alone is better in the case of subtitles. Here HAN decoder produces mistakes by repeating past translated personal pronouns. Subtitles is a challenging corpus for personal pronoun disambiguation because it usually involves dialogue between multiple speakers. It is defined as the ratio between the number of repeated and lexically similar content words over the total number of content words in a target document. The lexical similarity is obtained using WordNet. In some cases, HAN decoder achieves the best score because it produces a larger quantity of repetitions than other models. Although HAN boosts lexical cohesion, the scores are still far from the human reference, so there is room for improvement in this aspect. For coherence, we use a metric based on Latent Semantic Analysis (LSA) ( LSA is used to obtain sentence representations, then cosine similarity is calculated from one sentence to the next, and the results are averaged to get a document score. The joint HAN model consistently obtains the best coherence score, but close to other HAN models. Most of the improvement comes from the HAN decoder.