Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes
Table 4: Perplexity score (PPL) comparison on the PTB dataset and the AWD model. We use two different values for the batch size K --- the original one with K=20, and a prime one with K=19. The results directly corroborate the observation portrayed in Figure 4, where the obtained score is related to the diversity of grayscale values in each row.
['Experiment', 'K=20', 'K=19']
We compare the scores of a prime batch size K=19 with the scores of the original batch size K=20 for the AWD model with Alleviated TOI (P). When using a prime batch size, we observe consistent and increasing results as P increases. With the original batch size K=20, we observe a strong performance for P=7, but a low performance for P=10. The matrix with P=7 shows a good distribution---corresponding to the strong performance---and the matrix with P=10 shows that each row contains a low diversity of data points.