Training a Code-Switching Language Model with Monolingual Data
Table 1: ZH, EN, CS-PPL, CSP-PPL and overall perplexity on testing set
['[BOLD] Without normalization', '[BOLD] Without normalization', '[BOLD] Without normalization', '[BOLD] Without normalization', '[BOLD] Without normalization', '[BOLD] Without normalization']
Baselines in rows (a)(d)(g) represent the language model trained without constraints or normalization. , we observe that learning with pseudo-CS sentences indeed helps considerably in CS perplexity, which is reasonable because the LM has seen CS cases during training even though the training data is synthetic. However, comparing rows (b)(c) with (d) and (g) reveals that after applying additional constraints, the LM trained on monolingual data only is comparable or even better in terms of both monolingual (ZH and EN columns) and CS (CS-PPL and CSP-PPL columns) perplexity than LMs trained with pseudo-CS data. Whether using monolingual or pseudo-CS data for training, normalizing the output projection matrix generally improves language modeling. Even trained with monolingual data only, normalization also improves CSP-PPL, as shown in rows (a) and (j). Thus we conclude that the monolingual data in our corpus has a similar sentence structure, and normalization yields a similar latent space, aiding in switching between languages. After applying SKLD and normalization together, the CSP-PPL improves, yielding the best results in the monolingual data training case. The perplexity of CS points is reduced significantly when constraints are applied on the output projection matrix by minimizing SKLD or CD without degrading the performance on monolingual data. Rows (k)(n)(q) also show that combining the SKLD constraint with normalization results in the best performance on each kind of perplexity over only monolingual and pseudo-CS data. In addition to numerical analysis, we seek to determine if the overlapping level of embedding space is aligned with the perplexity results. We applied principal component analysis (PCA) on the output projection matrix, and then visualized the results on a 2-D plane. Fig. Fig. After synthesizing pseudo-CS data for training as shown in Fig. In Fig.