[
Table 7: Varying edit distance cost for SMS (small) dataset. Learning character pair alignment probabilities
['String feature', 'Precision', 'Recall', 'F-measure']
For each word in our vocabulary, we found the 100 closest pairs, where closeness is defined by our similarity function as described in Eq. We created a list of candidate word pairs by pairing every word with every other word in the cluster of 100 closest words. We take each Roman Urdu word as a sequence of Roman characters and its original Urdu script as a sequence of Urdu characters. We learn the alignment between the above two character-sequences in two different ways. First, we apply GIZA++ and learn the alignment with the Expectation-Maximization (EM) algorithm. Second, we implemented an unsupervised transliteration mining tool, details see Sajjad et al. Here, GIZA++ considers every word pair in the list of candidate pairs as a correct word pair to learn character alignments, whereas the transliteration mining tool penalizes the pairs that are less likely to be transliterations of each other during the training process. Since our list of candidate pairs is a mix of correct and incorrect pairs, the character alignments learned by the transliteration miner are likely to be better. The edit distance cost for each pair of characters can be computed from character alignments as cost(chari, charj)=|1−P(chari,charj)|. Our string similarity function uses these edit distance costs instead of manually defined costs. The F-measure of the cost learned by the miner and GIZA++ is competitive with the manually defined cost. edistgiza is affected by the noise in the data, which can be seen in its low precision compared to other methods. edistminer achieved the highest precision, though it has the lowest recall.