Competing Topic Naming Conventions in Quora: Predicting Appropriate Topic Merges and Winning Topics from Millions of Topic Pairs
Table 7: Evaluation results for Question 1.
['Evaluation Methods', 'Merge', 'Neighbor', 'Unmerge']
We observe that humans performed decently with majority voting accuracy of 0.75 for identification of merge classes. For the non-merge cases, humans performed very poorly (worst performance for the neighborhood class). These results indicate that the task of merging topics is difficult for Quora users as they seem to be confused by the non-merge cases in which they performed very poorly. Since the number of participants answering Question 1 for all the questions is 3, the micro-averaging and macro-averaging yields the same result. Overall, we observe that the regular Quora users find it difficult to judge if two given topics should merge or not, and in deciding the direction of merge.