AVA: an Automatic eValuation Approach to Question Answering Systems
Table 7: System-wise evaluation on TREC-QA and WikiQA using AVA model, A2((r,q∘t),(t,q∘r)).
['[BOLD] Metrics', '[BOLD] Metrics', '[BOLD] Kendall  [ITALIC] τ', '[BOLD] Kendall  [ITALIC] p', '[BOLD] RMSE\xa0±\xa0 [ITALIC] σ']
We note that (i) setting the right threshold on the dev. set, the error on P@1 is 0; (ii) this is not the case for MAP, which is a much harder value to predict as it requires to estimate an entire ranking; (iii) on the TREC-QA test set, AVA has an error ranging from 2 to 4.1 points on any measure; (iv) on the WikiQA test set, the error is higher, reaching 10%, probably due to a larger complexity of the questions; (v) the std. dev. is low, suggesting that AVA can be used to estimate system performance.