Leveraging Twitter for Low-Resource Conversational Speech Language Modeling
TABLE I: Language Vocabulary Size
['[BOLD] Language', '[BOLD] Types', '[BOLD] Tokens']
The in-domain data used in the experiments in this paper comes from the IARPA Babel program. This program focuses on keyword search for low-resource languages. The languages are low resource in the sense that they have fewer native speakers than the languages receiving the most attention from researchers and also in the sense that the provided training data is small in comparison to what is typically used. We exclusively focused on the so-called limited language pack, which consists of only ten hours of recorded telephone conversations. Our experiments were conducted using the Bengali, Tamil, Turkish and Zulu languages. As a point of comparison, Tagalog, which was not used in our experiments, has one third as many vocabulary items as Tamil. A large amount of data was also collected for Zulu and Tamil although the perplexity reduction was not as large as it was for Bengali and Turkish. There are a few possible explanations for this outcome. Both Zulu and Tamil have larger vocabularies For Zulu, there is an overlap in the character set with English and many Zulu speakers also tweet in English. The official Babel data also has some English words in the Zulu vocabulary. These factors combine to make it difficult to prevent the Zulu Twitter data from being polluted by a large amount of English text and could explain the relatively small gain for Zulu. However, as we shall see in the next section, the Zulu data is quite useful in the class n-gram model.