Learning Spoken Language Representations withNeural Lattice Language Modeling
Table 2: Results of our experiments in terms of accuracy (%). Some audio files in ATIS are missing, so the testing sets of manual transcripts and ASR transcripts are different. Hence, we do not report the results for ATIS using manual transcripts. The best results obtained by using ASR output for each dataset are marked in bold.
['Manual', '(a)', 'biLSTM', '[BOLD] ATIS -', '[BOLD] SNIPS 97.00', '[BOLD] SWDA 71.19', '[BOLD] MRDA 79.99']
All reported numbers are averaged over at least three training runs. Rows (a) and (b) can be considered as the performance upperbound, where we use manual transcripts to train and evaluate the models. We also use BERT-base Devlin et al. as a strong baseline, which takes ASR 1-best as the input (row (g)). Compare with the results on manual transcripts, using ASR results largely degrades the performance due to recognition errors, as shown in rows (e)-(g). In addition, adding pre-trained ELMo embeddings brings consistent improvement over the biLSTM baseline, except for SNIPS when using manual transcripts (row (b)). The baseline models trained on ASR 1-best are also evaluated on lattice oracle paths. We report the results as the performance upperbound for the baseline models (rows (c)-(d)).