NIT-Agartala-NLP-Team at SemEval-2020 Task 8:Building Multimodal Classifiers to tackle Internet Humor
Table 5: Annotator Bias Experiment
['Annotators', 'Sentiment Analysis F1 (%)', 'Sentiment Analysis Acc.(%)', 'humor F1(%)', 'humor Acc.(%)', 'Sarcasm F1(%)', 'Sarcasm Acc.(%)', 'Offensive F1(%)', 'Offensive Acc.(%)', 'Motivational F1(%)', 'Motivational Acc.(%)']
To address these concerns, we perform an experiment that randomly samples 100 memes from the training dataset (previously annotated) and gets four independent annotators to tag them on all five categories — sentiment, humor, sarcasm, offensive and motivational. The annotation process was mainly carried out in-house by the authors of this manuscript. We use basic guidelines (in the form of dictionary definitions and examples from the dataset) on tagging the categories and labels. We then calculate macro F1 scores and accuracy for each annotator and compare their performance with the Logistic Regression model’s predictions on the same 100 memes. We also calculate inter-annotator agreement metrics to understand better if all the annotators are on the same page. On calculating \newciteRandolph:2005 free-marginal multi-rater Kappa metrics, we found poor agreement on annotations over all the categories — Sentiment (0.22), humor (0.10), Sarcasm (0.09), Offense (0.36), Motivational (0.38). The poor agreement scores are an indicator of how concepts such as humor and offense can be subjective. We also find that the human annotators barely outperform our Logistic Regression model, which are indicators of low human performance in such tasks. Few Annotators also report different memes as random or nonsensical. On manual checking, we found these memes to be related to pieces of media (e.g., Star Wars, Lord of the Rings) that the annotator was not familiar with.