Identifying Compromised Accounts on Social Media Using Statistical Text Analysis
Table 3. Accuracy for different features and their combinations. Our method (LM) is compared to general text representations.
['%', '[ITALIC] COUNT', '[ITALIC] TF*IDF', '[ITALIC] Doc2Vec', '[ITALIC] Doc2Vec +  [ITALIC] TF*IDF', '[ITALIC] LM', '[ITALIC] LM +  [ITALIC] TF*IDF', '[ITALIC] LM +  [ITALIC] Doc2Vec', 'all']
It can be seen that Doc2Vec performs better than the word based models. This might be due to the fact that Doc2Vec is able to learn a better representation for each document, compared to TF*IDF and COUNT. Doc2Vec outperforms the TF*IDF model by up to 15 percentage points (≈27% relative improvement), if the amount of compromised tweets reaches 50%. The performance improvement is less drastic when only small amounts of tweets are compromised. We further find that LM performs best, as it outperforms TF*IDF and Doc2Vec with 6 and 7 percentage points when only 5 percent of tweets are compromised, respectively. The most distinguishing performance is achieved when the amount of compromised tweets reaches 50%. There, LM outperforms Doc2Vec with 9 percentage points (≈13% relative improvement) and TF*IDF with 24 percentage points (≈43% relative improvement). We argue that this is strong evidence for the superiority of the LM approach compared to other text classification features.