NRC-Canada at SMM4H Shared Task: Classifying Tweets Mentioning Adverse Drug Reactions and Medication Intake
Table 5: Task 1: Results of our best system (submission 1) on the test set when one of the feature groups is removed.
['[BOLD] Submission', '[ITALIC] Pclass1', '[ITALIC] Rclass1', '[ITALIC] Fclass1']
To investigate the impact of each feature group on the overall performance, we conduct ablation experiments where we repeat the same classification process but remove one feature group at a time. Comparing the two major groups of features, general textual features (row b) and domain-specific features (row c), we observe that they both have a substantial impact on the performance. Removing one of these groups leads to a two percentage points drop in Fclass1. The general textual features mostly affect recall of the ADR class (row b) while the domain-specific features impact precision (row c). Among the general textual features, the most influential feature is general-domain word embeddings (row b.2). Among the domain-specific features, n-grams generalized over domain terms (row c.1) and domain word embeddings (row c.3) provide noticeable contribution to the overall performance. In the Appendix, we provide a list of top 25 n-gram features (including n-grams generalized over domain terms) ranked by their importance in separating the two classes.