Character-level Transformer-based Neural Machine Translation
Table 1: Encoder and decoder parameters of the investigated models. At the encoder side, the models utilize 200 filters of width 1, 200 filters of width 2 etc. dff corresponds to the inner-layer has dimensionality. dm corresponds to the dimensionality of input and output. dk,dv correspond to the dimensionality of keys and values for attention heads, respectively.
['Encoder  [BOLD] Param.', 'Encoder  [BOLD] Transformer', 'Encoder  [BOLD] CharTrans.']
In this work, we compare three character-level and one subword-level NMT systems. First, we report results for the character-level model proposed by Lee et al. In this model, the decoder consists of two unidirectional GRU layers and the attention score is computed by a single-layer feedforward network. The encoder part implements an efficient source length reduction technique (detailed below), and adds a single-layer, bidirectional GRU on top. Second, we train a character-level Transformer and a subword-level Transformer Vaswani et al. And finally, we apply the source length reduction technique to the Transformer and build CharTransformer. We implemented this model in PyTorch Paszke et al. Layer sizes of the models are kept maximally comparable. Below, we highlight the important details of the models.