Question Answering on Scholarly Knowledge Graphs
Table 4: Performance comparison of different deep learning models on the task of question answering with different model sizes and architectures using the ORKG-QA benchmark dataset. The results suggest that different models perform differently on various question types, and generally the bigger the model the better it performs. For each question type, the best results are highlighted.
['[EMPTY]', '[BOLD] Questions type', '[BOLD] Precision @K #1', '[BOLD] Precision @K #3', '[BOLD] Precision @K #5', '[BOLD] Precision @K #10', '[BOLD] Recall @K #1', '[BOLD] Recall @K #3', '[BOLD] Recall @K #5', '[BOLD] Recall @K #10', '[BOLD] F1-Score @K #1', '[BOLD] F1-Score @K #3', '[BOLD] F1-Score @K #5', '[BOLD] F1-Score @K #10']
Experiment 2 - Different models of QA and their performance. We evaluate different types of QA models simultaneously to show the difference in performance metrics, execution time, and resource usage. JarvisQA’s QA engine employs the BERT L/U/S2 model due to its execution time and overall higher accuracy at higher positions.