A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking
Table 7: Evidence extraction: ranking setting
['model', 'precision @5', 'recall @5']
To evaluate the performance of the models in the ranking setup, we measure the precision and recall on five highest ranked ETS sentences (precision @5 and recall @5), similar to the evaluation procedure used in the FEVER shared task. The rankingESIM Hanselowski et al. The Tf-Idf model Thorne et al. We also evaluate the performance of DecompAttent and a simple BiLSTM Hochreiter and Schmidhuber To adjust the latter two models to the ranking problem setting, we used the hinge loss objective function with negative sampling as implemented in the rankingESIM model. As in the FEVER shared task, we consider the recall @5 as a metric for the evaluation of the systems. The Tf-Idf model reaches best results in terms of precision. The rankingESIM reaches a relatively low score and is not able to beat the random baseline. We assume this is because the model has a large number of parameters and requires many training instances.