RC-QED: Evaluating Natural Language Derivationsin Multi-Hop Reading Comprehension
Table 2: Ratings of annotated NLDs by human judges.
['# steps', 'Reachability', 'Derivability Step 1', 'Derivability Step 2', 'Derivability Step 3']
The evaluation results shown in Table 2 indicate that the annotated NLDs are of high quality (Reachability), and each NLD is properly derived from supporting documents (Derivability). [CONTINUE] On the other hand, we found the quality of 3-step NLDs is relatively lower than the others. [CONTINUE] Crowdworkers found that 45.3% of 294 (out of 900) 3-step NLDs has missing steps to derive a statement.