Analyzing the Perceived Severity of Cybersecurity Threats Reported on Social Media
Table 7: Model performance of identifying severe threats (CVSS scores ≥ 7.0) with Precision@k and area under the precision-recall curve (AUC) metrics. For majority random baseline, we average over 10 trails.
['[EMPTY]', 'P@10', 'P@50', 'P@100', 'AUC']
We evaluate our models’ performance at identifying severe threats five days ahead of the NVD publication date, within their top k predictions. We observe that tweet volume performs better than a random baseline; having a large number of tweets beforehand is a good indicator for high severity, however our approach which analyzes the content of messages discussing software vulnerabilities achieves significantly better performance; 86% of its top 50 forecasts were indeed rated as HIGH or CRITICAL severity in the NVD.