Multimodal Social Media Analysis for Gang Violence Prevention
Table 2. Results for detecting the psychosocial codes: aggression, loss and substance use. For each code we report precision (P), recall (R), F1-scores (F1) and average precision (AP). Numbers shown are mean values of 5-fold cross validation performances. The highest performance (based on AP) for each code is marked with an asterisk. In bold and red we highlight all performances not significantly worse than the highest one (based on statistical testing with 95% confidence intervals).
['[BOLD] Modality', '[BOLD] Features', '[BOLD] Fusion', '[BOLD] Aggression P', '[BOLD] Aggression R', '[BOLD] Aggression F1', '[BOLD] Aggression AP', '[BOLD] Loss P', '[BOLD] Loss R', '[BOLD] Loss F1', '[BOLD] Loss AP', '[BOLD] Substance use P', '[BOLD] Substance use R', '[BOLD] Substance use F1', '[BOLD] Substance use AP', '[BOLD] mAP']
Our results indicate that image and text features play different roles in detecting different psychosocial codes. Textual information clearly dominates the detection of code loss. We hypothesize that loss is better conveyed textually whereas substance use and aggression are easier to express visually. Qualitatively, the linguistic features with the highest magnitude weights (averaged over all training splits) in a linear SVM bear this out, with the top five features for loss being i) free, ii) miss, iii) bro, iv) love v) you; the top five features for substance use being i ) smoke, ii) cup, iii) drank, iv) @mention v) purple; and the top five features for aggression being i) Middle Finger Emoji, ii) Syringe Emoji, iii) opps, iv) pipe v) 2017. The loss features are obviously related to the death or incarceration of a loved one (e.g. miss and free are often used in phrases wishing someone was freed from prison). The top features for aggression and substance use are either emojis which are themselves pictographic representations, i.e. not a purely textual expression of the code, or words that reference physical objects (e.g. pipe, smoke, cup) which are relatively easy to picture.