Design Challenges in Named Entity Transliteration
Table 3: T2T 1-Best Word Error Rate (WER) when filtering single occurrence name tokens from training and/or testing.
['[BOLD] Dataset', '[BOLD] Full Train, Full Test (Baseline)', '[BOLD] Full Train, Filtered Test', '[BOLD] Filtered Train, Full Test', '[BOLD] Filtered Train, Filtered Test']
One design question that we set out to answer is how to be handle unique name token pairs in dataset curation (e.g., the ‘EN-KA’ transliteration pair [“escalada”, “エスカラーダ”] occurs only once in the data we collected from Wikidata). Name tokens, like other language tokens, follow a Zipfian distribution and thus there are likely to be many tokens that appear infrequently. On the one hand, tokens that occur only once may be erroneous, and it may be better to exclude them from the training and test sets. Taking the English to Hebrew dataset as an example, 77% of the name tokens only occur once. Our hypothesis was that given this distribution of token frequency, excluding them would likely remove more useful information than noise. All of these results use the Transformer neural approach with 128 units per layer and 2 hidden layers. If we also filter the tokens from train (as shown in column 4), we see an increase in WER. The worse WER is achieved when filtering the train set but testing on the full test set. In English to Hebrew (WD-EN-HE), filtering the train set but testing on the full set increases the WER by 7.0 points over the baseline. Industrial transliteration systems will contain dictionary look-ups for frequent named entity transliterations and employ automated methods, as evaluated here, to handle out-of-vocabulary cases. Thus, it stands to reason that a fair evaluation of automated methods should include infrequent names even in the presence of some noise. The results presented in other sections include unique tokens at both train and test time.