See and Read: Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Data
Table 5: Best F1 results for each modality. All results are for the observation period of 212 days.
['[BOLD] Model', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F1', '[BOLD] Architecture']
Textual models usually performed better than visual models in terms of F1 score. The best visual result from ResNeXt is not surprising as the pretrained weights were trained with 940 million Instagram images. However, visual models, as previously discussed, usually provided better precision scores, while textual models had higher recall scores. Considering the fused visual and textual features — for the deep learning models —, we achieve almost equivalent scores using ELMo concatenated with any ResNet, and ResNeXt architectures, where the best F1 score (0.79) was achieved with ELMo + ResNet-34. For the feature engineering dataset, however, the F1 score was not improved as expected when using fused features, resulting in a worse F1 score (0.73, for 212 days). This result can be related to the difference of features when concatenating both modalities, since we have 64 textual features and 12 visual features disposed into different representational spaces. It also indicates the necessity of more investigation on how to fuse modalities when using feature engineering, as previously explored in other studies [morales2018linguistically].