Actionable and Political Text Classification using Word Embeddings and LSTM
Table 1: LSTM Language Models for Actionability. (V: Vocabulary Size)
['[BOLD] Language', '[BOLD] Training Samples', '[BOLD] Test Samples', '[BOLD] Training Accuracy ( [ITALIC] V=100k)', '[BOLD] Test Accuracy ( [ITALIC] V=100k)', '[BOLD] Training Accuracy ( [ITALIC] V=20k)', '[BOLD] Test Accuracy ( [ITALIC] V=20k)']
We examine actionability across languages in our experiments. A pair of training and test datasets is therefore created for each language, as well as one dataset that includes all languages. Some languages have sparse data and smaller number of samples, while others are larger. Our dataset sizes vary from 27k messages for the language with the least data (Farsi), to 8.5 million messages for the language with the most (English). A combined language dataset is also prepared, which contains 1.7 million sampled messages across all languages. Languages such as Czech (cs), Finnish (fi), Slovak (si) and Swedish (sv) show the highest accuracies of over 90%. The lowest accuracies are for languages such as Arabic (ar), Farsi (fa) and Thai (th), where the accuracies are slightly below 80%, though in some cases like fa and th, the low accuracy may be attributed to sparsity of data. As expected, certain languages are more predictable in terms of actionability than others, and we observe a variation of 15% between the least and most accurate models. In our experiments we consider two vocabulary sizes for each language. The first is a smaller vocabulary of size 20,000, and the second is a larger one of size 100,000. The larger vocabulary adds significant overhead in training time, since more params need to be learnt by the model.