Gmail Smart Compose: Real-Time Assisted Writing
Table 4. Results using the Seq2Seq approach.
['Model', '# Params', 'Training Time (h)', 'Test Log Perplexity']
In our baseline seq2seq model, both encoder and decoder have two 1024-dimensional LSTM layers. We can see that the LSTM seq2seq model outperforms the LSTM language models with similar number of model params, indicating that an encoder-decoder architecture with attention is more effective in modeling the context information. Along this line of experiments, we also evaluated how well the state-of-the-art seq2seq models from neural machine translation can perform on the Smart Compose task. Specifically, we selected Transformer Big model from Vaswani et al. BiLSTM-6-LSTM-8-1024 is the RNMT+ model, where there are 6 bidirectional LSTM layers in the encoder and 8 unidirectional LSTM layers in the decoder. Without any parameter tuning, both Transformer Big model and RNMT+ perform fairly well on the Smart Compose data, with perplexity on par with each other and better than our LSTM seq2seq model baseline.