Simple Fusion: Return of the Language Model
Table 7: Perplexity and average entropies of the distributions generated by our systems on the English-Turkish dev set.
['[BOLD] Method', '[BOLD] Perplexity', '[BOLD] Average entropy']
S6SS0SSS0Px4 Impact on the TM distribution With the PostNorm strategy, the TM still produces a distribution over the target vocabulary as the scores are normalized before the combination with the LM. This raises a natural question: How different are the distributions generated by a TM trained under PostNorm loss from the distributions of the baseline system without LM? Tab. As expected, the RNN-LM has higher perplexity than the baseline as it is a weaker model of translation. The RNN-LM also has a higher average entropy which indicates that the LM distributions are smoother than those from the baseline translation model. The TM trained under PostNorm loss has a much higher perplexity which suggests that it strongly relies on the LM predictions and performs poorly when it is not combined with it. However, the average entropy is much lower (1.82) than both other models, i.e. it produces much sharper distributions.