Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension
Table 2: Experiment results for state-of-the-art models demonstrating degrading performance under spoken data. All models were trained on the full SQuAD training set. Mnemonic Reader and FusionNet are denoted by Mreader and F-NET, respectively. SQuAD dev set and Spoken SQuAD testing set are denoted by SQuAD-dev and SpokenS-test, respectively.
['[BOLD] MODEL', '[BOLD] SQuAD-dev EM', '[BOLD] SQuAD-dev F1', '[BOLD] SpokenS-test EM', '[BOLD] SpokenS-test F1']
We trained five reading comprehension models mentioned in Section 3.1 on the SQuAD training set, and they were tested on SQuAD dev set and Spoken SQuAD testing set. The number of questions in the Spoken SQuAD testing set is less than the original SQuAD dev set because some of the examples are removed as mentioned in Section 2.1. To make the comparison fair, on SQuAD dev set, we only report the results on the questions also in Spoken SQuAD testing set. The average F1 score fell to 55.4% when there are ASR errors. Similar phenomenon is observed on EM. The impact of ASR errors is significant for machine comprehension models.