Identifying Compromised Accounts on Social Media Using Statistical Text Analysis
Table 1. Ablation study using different measures.
['Measure', 'All', 'Max', 'Min', 'Mean', 'Var.']
In this experiment 50% of the tweets of a compromised account are compromised. We observe that we gain maximum performance over all metrics when all features are utilized (see column “All”). The classifier reaches an accuracy of 0.80, high precision (0.90) and Recall at 0.68. Both measures are combined in the F1-score, which reaches 0.78. We would like to note that a precision of 0.90 can be sufficient for many practical applications, where the system could alert users or platform providers. When features are investigated individually, we find that the features “Max” and “Variance” perform worst. The features “Min” and “Mean” perform almost equally in isolation and much better than the other two. We find the minimum sample of the two probability distributions to be a strong signal. This further confirms our earlier observations that compromised accounts can be distinguished by their higher average KL-divergence.