REST: A Thread Embedding Approach for Identifying and Classifying User-specified Information in Security Forums
Table 8: Identification Precision: the precision of the identified thread of interest with the similarity-based method.
['[EMPTY]', 'OffensComm.', 'HackThisSite', 'EthicHack', 'Avg.']
Estimating precision. To evaluate precision, we want to identify what percentage of the retrieved threads are relevant. To this end, we resort to manual evaluation. We have labeled 300 threads from each dataset retrieved with 50% of the keywords and we get our annotators to identify if they are relevant. We understand that on average more than 97.5% of the threads identified with the similarity based method are relevant with an inter-annotator agreement Fleiss-Kappa coefficient of 0.952.