Question Answering on Scholarly Knowledge Graphs
Table 3: JarvisQA performance on the ORKG-QA benchmark dataset of tabular data. The evaluation metrics are precision, recall, and F1-score at k position. JarvisQA is compared against two baselines on the overall dataset and specific question types. The symbol (-) indicates that the performance metric showed no difference than the reported value for higher K values. The results suggest that JarvisQA outperforms the baselines by 2-3 folds.
['[BOLD] Questions type', '[BOLD] Baseline', '[BOLD] Precision @K #1', '[BOLD] Precision @K #3', '[BOLD] Precision @K #5', '[BOLD] Precision @K #10', '[BOLD] Recall @K #1', '[BOLD] Recall @K #3', '[BOLD] Recall @K #5', '[BOLD] Recall @K #10', '[BOLD] F1-Score @K #1', '[BOLD] F1-Score @K #3', '[BOLD] F1-Score @K #5', '[BOLD] F1-Score @K #10']
In order to evaluate the performance of JarvisQA, we run the system and other baselines on the ORKG-QA dataset at various k values (k denotes the position of the correct answer among all retrieved answers). For this experiment we evaluate k∈{1,3,5,10}. Moreover, the experiment was conducted on a specific subset of questions (based on types) to show the performance of the system for certain categories of questions. The tested question categories are: Normal: normal questions about a specific cell in the table with a direct answer; Aggregation: questions about aggregation tasks on top of the table; Related: questions that require retrieving the answer from another cell in the table; Similar: questions that address the table using similar properties (e.g., synonyms). The results show that JarvisQA performs better by 2-3 folds against Lucene, and Random baselines respectively.