A Question-Entailment Approach to Question Answering
Table 5: Inter-Annotator Agreement (IAA) over all ratings in the manual evaluation of the retrieved answers. Partial IAA over two ratings "Correct" and "Incorrect".
['[BOLD] Assessors', '[BOLD] IAA  [BOLD] P (%)', '[BOLD] IAA  [BOLD] F1 (%)', '[BOLD] Partial IAA  [BOLD] P (%)', '[BOLD] Partial IAA  [BOLD] F1 (%)']
In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score. As there are no negative labels (only true or false positives for each category), Recall is 100%. We also computed a partial IAA by grouping the "Correct and Complete Answer" and "Correct but Incomplete" ratings (as Correct), and the "Incorrect but Related" and "Incorrect" ratings (as Incorrect). The average agreement on distinguishing the Correct and Incorrect answers is 94.33% F1 score. Therefore, we used the evaluations performed by assessor A for both methods. The official results of the TREC LiveQA track relied on one assessor per question as well.