ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations
Table 6: Pearson’s r×100 on STS2017 with different mega-batch sizes M.
['[ITALIC] M', 'Word', 'Trigram', 'LSTM']
Therefore, we propose a procedure we call “mega-batching.” We aggregate M mini-batches to create one mega-batch and select negative examples from the mega-batch. Once each pair in the mega-batch has a negative example, the mega-batch is split back up into M mini-batches and training proceeds. For all models, larger mega-batches improve performance. There is a smaller gain when moving from 20 to 40, but all models show clear gains over M=1.