1 Introduction
Table 1: Classification results for four languages (German, English, Spanish, and French). TL indicates the total number of labeled messages, while Pos. indicates the percentage of those who were labeled as flood-related. P, R, and F indicate Precision, Recall, and F-Measure respectively. We report the performance of a mono-lingual classifier, of a cross-language classifier with “cold start” (uses no training data in the target language), and of a cross-language classifier with “warm start” (uses 300 labeled items in the target language).
['[EMPTY]', '[EMPTY]', '[EMPTY]', 'Glove embeddings mono-lingual', 'Glove embeddings mono-lingual', 'Glove embeddings mono-lingual', 'Glove embeddings cold-start', 'Glove embeddings cold-start', 'Glove embeddings cold-start', 'Glove embeddings warm-start', 'Glove embeddings warm-start', 'Glove embeddings warm-start', 'MUSE embeddings mono-lingual', 'MUSE embeddings mono-lingual', 'MUSE embeddings mono-lingual', 'MUSE embeddings cold-start', 'MUSE embeddings cold-start', 'MUSE embeddings cold-start', 'MUSE embeddings warm-start', 'MUSE embeddings warm-start', 'MUSE embeddings warm-start']
Training data was labeled using crowdsourcing platform GetHybrid (https://gethybrid.io/). The amount of labeled data comprises over 7,000 annotated messages, each one annotated by three annotators independently. Four training datasets were created, containing labeled tweets in German, English, Spanish, and French. This amount of data is typical in automatic classification tasks \parenciteimran2015processing. In the mono-lingual experiment we simply use labeled data in one language to predict the label for messages in the same language. In the cold-start experiment we train a classifier for a new language using only labeled data for other languages; for instance for automatically labeling tweets in Spanish, we use a classifier trained on labeled data for English, German, and French. In the warm-start experiment we use a set-up similar to the one of the cold-start experiment, but we add a limited number of messages (300) labeled in the target language. Each experiment is done once using GloVe embeddings and once using MUSE embeddings. We also include the total number of tweets manually labeled by crowdsourcing workers (TL) and its percentage of positive (Pos.) tweets. Collections for German, English, Spanish, and French consider tweets posted during floods happening in the last two years in Germany, the UK, Spain and Mexico, and France respectively. Hence, the number of labeled tweets and the percentage of positive examples differ across languages.