NAACL’19 Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
Table 3: Quality of different baselines on different sub-tasks (Section 5). All the numbers are in percentage. Top machine baselines are in bold.
['Setting', 'Target set', 'System', '[ITALIC] Pre.', '[ITALIC] Rec.', '[ITALIC] F1']
To measure the performance of the IR system, we use the index containing Up. Given each claim, we query the top k perspectives, ranked according to their retrieval scores. We tune k on our development set and report the results on the test section according to the tuned parameter. We use IR results as candidates for other solvers (including humans). In order to train BERT on this task, we use the IR candidates as the training instances. We then tune a threshold on the dev data to select the top relevant perspectives. In order to measure human performance, we create an interface where two human annotators see IR top-k and select a minimal set of perspectives (i.e., no two equivalent perspectives).