Building Chatbots from Forum Data:Model Selection Using Question Answering Metrics
Table 3: Number of good answers according to manual annotation of the answers to 50+50 questions by the three models from Table 1.
['[EMPTY]', '[BOLD] Optimizing for', '[BOLD] #  [ITALIC] Good [BOLD]  answers according to  [BOLD] Ann. 1', '[BOLD] #  [ITALIC] Good [BOLD]  answers according to  [BOLD] Ann. 2', '[BOLD] #  [ITALIC] Good [BOLD]  answers according to  [BOLD] Ann. 3', '[BOLD] #  [ITALIC] Good [BOLD]  answers according to  [BOLD] Ann. 4', '[BOLD] Avg.']
The average of the four annotators suggests that optimizing for MAP yields the best overall results. Note that all systems perform slightly worse on the second set of questions. This should be expected as the latter are different from those used for training the models. Overall, the MAP-based system appears to be more robust, with only 2.25 points absolute decrease in performance (compared to 5 and 4.75 for the systems using BLEU and seq2seq loss, respectively).