Large Scale Question Answering using Tourism Data
Table 7: Performance of different systems including the CsrQA model on our task as measured using human judgements (Human Scores) and gold-reference data (Machine Scores). Accuracy reported in %.
['[EMPTY]', '[BOLD] Human Scores', '[BOLD] Machine Scores']
As can be seen from the table, the absolute performance of the systems as measured by the human annotators is higher indicating the presence of false negatives in the dataset. We compute pair-wise correlation coefficients between CsrQA, CsQA and CrQA, and we find there is moderately positive correlation correlation with high confidence between the human judgements and gold-data based measurements for both Acc@3 (¯ρ=0.39, p-value<0.0009) as well as on Acc@5 (¯ρ=0.32 p-value<0.04). Please see appendix for more details.