Speech Recognition Challenge in the Wild: Arabic MGB-3
Table 2: word-level inter annotator disagreement on the development data across the four different human references before/after normalization (in %).
['[EMPTY]', 'ref2', 'ref3', 'ref4']
This indicates that there is about 13% disagreement between the annotators for the MGB-3 data. We will report results for the MGB-3 data after normalization only. Similar to MGB-2, we provided an open source baseline systems for the challenge for both the speech transcription and dialect identification tasks. Word Error Rate (WER) continues to be the most commonly used metric for evaluating ASR. Usually, transcription of the speech signal is deterministic and one manual transcription should be a sufficient reference. For the MGB-3 Challenge, we investigated using MR-WER. This metric is based on comparing the recognized text against multiple manual transcriptions of the speech signal, which are all considered valid references. This approach thus accepts a recognized word if any of the references include it in the same form.