Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes
Table 2: Perplexity score (PPL) comparison of the AWD model, on the three datasets, with batch sizes K=20 (PTB), K=80 (WT2) and K=60 (WT103), with different levels of Token Order Imbalance (TOI). With Alleviated TOI (P), we use a prime batch size of K=19 (PTB), K=79 (WT2) and K=59 (WT103).
['Experiment', 'PTB', 'WT2', 'WT103']
We use the test perplexity after the same equivalent number of epochs. The different Alleviated TOI (P) experiments use a different number of overlapped sequence: An Alleviated TOI (P) means building and concatenating P overlapped sequences. Our results indicate that an Alleviated TOI (P) is better than the Standard TOI, which is better than an Extreme or Inter-batch TOI. We note a tendency that higher values of P lead to better results, which is in accordance with our hypothesis that a higher TOI ratio (P−1)/P improves the results.