Large Scale Question Answering using Tourism Data
Table 2: QA Pairs in train,validation and test sets
['[EMPTY]', '[BOLD] #Ques.', '[BOLD] QA pairs', '[BOLD] Tokens per ques.', '[BOLD] #QA Pairs with Hotels', '[BOLD] #QA Pairs with Restr.', '[BOLD] #QA Pairs with Attr.']
We expect that our automated QA pair extraction methods are likely to have some degree of noise ( In order to facilitate accurate bench-marking, we crowd-source and clean our validation and test sets. We use the Amazon Mechanical Turk(AMT) Workers are presented with a QA-pair, which includes the original question, an answer-entity extracted by our rules and the original forum-post response thread where the answer entity was mentioned. Workers are then asked to check if the extracted answer entity was mentioned in the forum responses as an answer to the user question. We spend $0.05 for each QA pair costing a total of $550. The crowd-sourced cleaning was of high quality; on a set of 280 expert annotated question-answer pairs, the crowd had an agreement score of 97%.