Building Chatbots from Forum Data:Model Selection Using Question Answering Metrics
Table 1: Evaluation results using the seq2seq model and optimizing during training for MAP (on DEV) vs. BLEU (on DEV) vs. the seq2seq loss (on TRAIN). The following columns show some results on TEST when selecting the best training model on DEV (for MAP and BLEU) and on TRAIN (for the seq2seq loss). We report BLEU and MAP, as well as the iteration at which the best value was achieved on DEV/TRAIN, and the average length of the generated answers on TEST.
['[EMPTY]', '[BOLD] Optimizing for', '[BOLD] MAP', '[BOLD] BLEU', '[BOLD] Iteration', '[BOLD] Ans. Len.']
In a second experiment, we compared model selection strategies when optimizing for MAP (TF×IDF+qc-sim) vs. BLEU vs. seq2seq loss. We further report the results for a baseline for the SemEval2016 Task 3, subtask A Nakov et al. For each model selection criterion, we report its performance and statistics on the test dataset about the model that was best-performing on the development dataset. First, we randomly selected 50 questions from the test set of SemEval-2016 Task 3. However, we did not use the original questions, as they can contain multiple sentences and thus can be too long; instead, we selected a single sentence that contains the core of the question (and in some cases, we simplified it a bit further). We further created 50 new questions, which are more personal and conversational in nature, but are still generally related to Qatar. The answers produced by the three systems for these 100 questions were evaluated independently by four annotators, who judged whether each of the answers is good.