AVA: an Automatic eValuation Approach to Question Answering Systems
Table 9: Details of system-wise Evaluation on ADS benchmark dataset
['[BOLD] ADS Split', '[BOLD] Evaluator', '[BOLD] S1', '[BOLD] S2', '[BOLD] S3', '[BOLD] S4', '[BOLD] S5', '[BOLD] S6', '[BOLD] S7', '[BOLD] S8', '[BOLD] Kendall  [ITALIC] τ', '[BOLD] Kendall  [ITALIC] p', '[BOLD] RMSE\xa0±\xa0 [ITALIC] σ']
6.5.2 Results on ADS We use ADS dataset in this evaluation. The task is more challenging as AVA only receives one best answer for a system selected from different candidate pools. There was also no control of the sources for the candidates. We note a lower correlation due to the fact that the 8 evaluated systems have very close Accuracy. On the other hand, the RMSE is rather low 3.1% and the std. dev. is also acceptable <0.02, suggesting an error less than 7% with a probability > 95%.