Identifying Compromised Accounts on Social Media Using Statistical Text Analysis
Table 2. LM and Doc2Vec features, with different percentages of compromised tweets.
['%', '[ITALIC] LM Accuracy', '[ITALIC] LM  [ITALIC] F1', '[ITALIC] LM Precision', '[ITALIC] LM Recall', '[ITALIC] Doc2Vec Accuracy', '[ITALIC] Doc2Vec  [ITALIC] F1', '[ITALIC] Doc2Vec Precision', '[ITALIC] Doc2Vec Recall']
We turn to research question (IV) and investigate how our language model based approach compares to text classification features. Best performance is reached when 50% of tweets are compromised, with accuracy at 0.80 and 0.71, respectively. When the number of compromised tweets is reduced by half, the accuracy drops to 0.75 and 0.69. When the amount of compromised tweets is set to 10% and 5%, accuracy further reduces for both approaches. For random ratios, the performance is slightly lower than for the fixed 25% ratio. It is expected that this performance falls somewhere within the lower (5%) and upper (50%) bounds for performance. Summarizing, we see that LM achieves higher Accuracy and Precision, whereas Doc2Vec achieves higher Recall.