Sign Language Translation with Transformers
Table 3: G2T performance comparison of Transformers on RWTH-PHOENIX-Weather 2014T with different number of enc-dec layers.
['Layers', 'Dev Set BLEU-1', 'Dev Set BLEU-2', 'Dev Set BLEU-3', 'Dev Set BLEU-4', 'Dev Set ROUGE-L', 'Dev Set METEOR', 'Test Set BLEU-1', 'Test Set BLEU-2', 'Test Set BLEU-3', 'Test Set BLEU-4', 'Test Set ROUGE-L', 'Test Set METEOR']
To choose the best model, we will mainly take into account the BLEU-4 score, as it is the most widely used metric in machine translation currently. Though the model with 4 layers obtains better ROUGE score, this metric places more emphasis on recall and is more interpretable in summarization tasks. Moreover, a smaller model has the advantage of taking up less memory and computation time. A reason why the smaller model than the more common one with 6 layers is more effective for our task is likely because the German sign language glosses are more similar to spoken German compared to two different spoken languages such as English and German. Also, because our dataset is much smaller than those used in standard machine translation tasks, larger networks may be disadvantaged. Repeating the same experiment on ASLG-PC12, we also find 2 layers to be the optimal model size. We carry out the rest of our experiments on this dataset with networks of 2 enc-dec layers.