On NMT Search Errors and Model Errors: Cat Got Your Tongue?
Table 4: Length normalization fixes translation lengths, but prevents exact search from matching the BLEU score of Beam-10. Experiment conducted on 48.3% of the test set.
['[BOLD] Search', '[BOLD] W/o length norm.  [BOLD] BLEU', '[BOLD] W/o length norm.  [BOLD] Ratio', '[BOLD] With length norm.  [BOLD] BLEU', '[BOLD] With length norm.  [BOLD] Ratio']
We can find the global best translations under length normalization by generalizing our exact inference scheme to length dependent lower bounds γk. (e.g. zero to 1.2 times the source sentence length). (4) Exact search under length normalization does not suffer from the length deficiency anymore (last row in Tab. but it is not able to match our best BLEU score under Beam-10 search. This suggests that while length normalization biases search towards translations of roughly the correct length, it does not fix the fundamental modelling problem.