Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences
Table 3: Results of perplexity (PPL) on a valid set and test set for different training strategies. We report the overall PPL, and code-switching points (en-zh) and (zh-en), as well as the monolingual segments PPL (en-en) and (zh-zh).
['[BOLD] Training Strategy', '[BOLD] Overall  [BOLD] valid', '[BOLD] Overall  [BOLD] test', '[BOLD] en-zh  [BOLD] valid', '[BOLD] en-zh  [BOLD] test', '[BOLD] zh-en  [BOLD] valid', '[BOLD] zh-en  [BOLD] test', '[BOLD] en-en  [BOLD] valid', '[BOLD] en-en  [BOLD] test', '[BOLD] zh-zh  [BOLD] valid', '[BOLD] zh-zh  [BOLD] test']
Pointer-Gen consistently performs better than state-of-the-art models such as EC and SeqGAN. Comparing the results of models trained using only generated samples, (2a-2b) leads to the undesirable results that are also mentioned by pratapa2018language, but it does not apply to Pointer-Gen (2c). We can achieve a similar results with the model trained using only real code-switching data, rCS. This demonstrates the quality of our data generated using Pointer-Gen. In general, combining any generated samples with real code-switching data improves the language model performance for both code-switching segments and monolingual segments. Applying concatenation is less effective than the two-step training strategy. Moreover, applying the two-step training strategy achieves the state-of-the-art performance.