REST: A Thread Embedding Approach for Identifying and Classifying User-specified Information in Security Forums
Table 11: Classification: the performance of the five different methods in classifying threads in 10-fold cross validation.
['Datasets', 'Metrics', 'BOW', 'NMF', 'SWEM', 'FastText', 'LEAM', 'BERT', '[BOLD] REST']
REST compared to the state-of-the-art. Our approach compares favourably against the competition. REST outperforms other baseline methods with at least 1.4 percentage point in accuracy and 0.7 percentage point in F1 score, except BERT. First, using BERT “right out of the box” did not give good results initially. However, we fine-tuned BERT for this domain. BERT performs poorly on two sites, HackThisSite and EthicalHackers, while it performs well for OffensiveCommunity. We attribute this to the limited training data in terms of text size and also the nature of the language users use in such forums. For example, we found that the titles of two misclassified threads contained typos and used unconventional slang and writing structure “ Hw 2 gt st4rtd with r3v3r53 3ngin33ring 4 n00bs!!” , “metaXploit 3xplained fa b3ginners!!!”. We intend to investigate BERT and how it can be tuned further in future work.