Cseq2seq: Cyclic Sequence-to-Sequence Learning
Table 1: Case-insensitive BLEU scores on the Chinese-English translation tasks. #Param = the number of parameters in NMT model. MP and LS denote the mean-pooling and last-state respectively. ALL = total BLEU score on test sets. We highlight the best results in bold for each test set. “↑/⇑”: significantly better than Moses (p < 0.05/p < 0.01); “+/++”: significantly better than RNNSearch (p < 0.05/p < 0.01).
['[BOLD] System', '[BOLD] #Param', '[BOLD] MT05', '[BOLD] MT02', '[BOLD] MT03', '[BOLD] MT04', '[BOLD] MT06', '[BOLD] MT08', '[BOLD] ALL']
We find that the all NMT models achieve very promising results except the vanilla Seq2seq. This is mainly because that the vanilla Seq2seq model heavily relies on multi-layered deep encoders and decoders which are beyond our research scope. Our implemented RNNSearch model outperforms the Moses up to 3 points in total. Replacing the attention mechanism with GRU-based RNN model, Cseq2seq-I obtains another improvements of 1.94 points with the last-state strategy, and yields the best results across all test sets. Even without the attention mechanism, all Cseq2seq-II models outperform both RNNSearch and Moses system. All these improvements are significant. We then performed parameter sharing to verify whether there are parameters redundant in Cseq2seq-II. Firstly, we shared GRU parameters, which pruned 4.87M parameters. This suggests that different languages can indeed share their recurrent parameters, not only across the encoder or decoder Johnson et al. Secondly, we shared word embeddings, which pruned 18.6M parameters. Under this setting, “Cseq2seq-II + SWord” achieves 34.89 BLEU points, a gain of 0.28 BLEU points over Cseq2seq-II, indicating the two target-side word embeddings indeed share very similar syntactic properties and can be typed together. Finally, we combined the above two approaches, leading to a total of 23.47M parameters being pruned. With only 51.72M parameters, “Cseq2seq-II + SGRU + SWord” achieves 34.89 BLEU points, yielding 0.28 BLEU points improvement. In other words, we pruned 31% parameters in Cseq2seq-II with no performance loss, which will benefit applications in memory-limited devices.