A Comparison of Neural Models for Word Ordering
Table 3: BLEU scores for PTB word-ordering task for different search heuristics and beam sizes (test).
['[BOLD] Model', '[ITALIC] none', '[ITALIC] f(⋅)', '[ITALIC] g(⋅)']
Since decoding is slow for large beam sizes, we compare bag2seq to the n-gram and Rnnlm using a small beam of size 5 in Tab. The first three rows show that decoding without heuristics is much easier with bag2seq and outperforms n-gram and Rnnlm by a large margin with 33.4 BLEU. The Rnnlm needs heuristic f(⋅) to match this performance. For bag2seq, using heuristic estimates is worse than just using its partial scores for search. We suspect that its partial model scores are obfuscated by the heuristic estimates and the amount of their contribution should probably be tuned on a heldout set. Using the same beam size, ensembles yield better results but the best results are achieved by combining Rnnlm and bag2seq (37.9 BLEU). This confirms our findings on WMT data that these models are highly complementary for word ordering. The results for beam=64 follow this pattern and identify an interaction between heuristics and beam size. While we get the best results for beam=5 using f(⋅), heuristic g(⋅) seems to perform better for larger beams, perhaps because the internal unigram statistics become more reliable. Finally, Rnnlm+bag2seq with g(⋅) and beam=64 outperforms Lstm-512 by 0.8 BLEU. This is significant because decoding in this configuration is also ∼4x faster than decoding with a single Rnnlm and beam=512 as shown in Fig.