WERd: Using Social Text Spelling Variantsfor Evaluating Dialectal Speech Recognition
Table 3: WER vs. TER vs. WERd vs. MR-WER, after normalization (in %).
['[EMPTY]', '[BOLD] WER', '[BOLD] TER', '[BOLD] WERd', '[BOLD] MR-WER']
We first evaluated the ASR system on our two-hour dialectal Arabic test dataset using WER with respect to each of the five references. We can see that the WER is much higher on our dialectal Arabic dataset, ranging in 40–50%. TER and WERd scores calculated with respect to each of the five references and shows for both metrics a strong correlation with WER. We can also see that the scores for WERd are halfway between WER and MR-WER (e.g., for ref1, it is 34.3 vs. 46.2 and 25.3, respectively), but without the need for additional human references.