Question Answering by Reasoning Across Documents with Graph Convolutional Networks
Table 4: Accuracy and precision at K (P@K in the table) analysis overall and per query type. Avg. |Cq| indicates the average number of candidates with one standard deviation.
['[EMPTY]', '[BOLD] Relation', '[BOLD] Accuracy', '[BOLD] P@2', '[BOLD] P@5', '[BOLD] Avg. | [ITALIC] Cq|', '[BOLD] Supports']
In this section we provide an error analysis for our best single model predictions. First of all, we look at which type of questions our model performs well or poorly. There are more than 150 query types in the validation set but we filtered the three with the best and with the worst accuracy that have at least 50 supporting documents and at least 5 candidates. We observe that questions regarding places (birth and death) are considered harder for Entity-GCN. We then inspect samples where our model fails while assigning highest likelihood and noticed two principal sources of failure i) a mismatch between what is written in Wikipedia and what is annotated in Wikidata, and ii) a different degree of granularity (e.g., born in “London” vs “UK” could be considered both correct by a human but not when measuring accuracy).