Guided Alignment Training for Topic-Aware Neural Machine Translation
Table 4: Comparison of different loss functions and weight ratios for guided alignment.
['E-commerce En→Fr', 'BLEU %', 'TER %']
To balance decoder cost and the attention weight cost, we experimented with different weights for these costs. Besides fixing the cost ratios during training, we also apply a heuristic to adjust the ratio as the training is progressing. One approach is to set a high value for the alignment cost in the beginning, then decay the weight to 90% after every epoch, finally eliminating the influence of the alignment after some time. This approach helps for the IWSLT task, but not on the e-commerce task. We assume that the alignment for the TED talk sentences seems to be easier for NMT to learn on its own than the alignment between product titles and their translations. Since the cross-entropy function has a consistent form as decoder cost, we decided to use it in further experiments. We extracted the NMT attention weights and marked the connection with the highest score as hard alignment for each word. It can be seen from the graph that the guided alignment training truly improves the alignment correspondence.