Evaluating Semantic Parsing against a Simple Web-based Question Answering Model
Table 3: Results on development (average over random splits) and test set. Middle: results on all examples. Bottom: results on the subset where candidate extraction succeeded.
['[BOLD] System', '[BOLD] Dev  [BOLD] F1', '[BOLD] Dev  [BOLD] p@1', '[BOLD] Test  [BOLD] F1', '[BOLD] Test  [BOLD] p@1', '[BOLD] Test  [BOLD] MRR']
WebQA obtained 32.6 F1 (33.5 p@1, 42.4 MRR) compared to 40.9 F1 of CompQ. Our candidate extraction step finds the correct answer in the top-K candidates in 65.9% of development examples and 62.7% of test examples. Thus, our test F1 on examples for which candidate extraction succeeded (WebQA-Subset) is 51.9 (53.4 p@1, 67.5 MRR). Not using a KB, results in a considerable disadvantage for WebQA. KB entities have normalized descriptions, and the answers have been annotated according to those descriptions. We, conversely, find answers on the web and often predict a correct answer, but get penalized due to small string differences. E.g., for “what is the longest river in China?” we answer “yangtze river”, while the gold answer is “yangtze”. To quantify this effect we manually annotated all 258 examples in the first random development set split, and determined whether string matching failed, and we actually returned the gold answer. This improved performance from 53.6 F1 to 56.6 F1 (on examples that passed candidate extraction). Further normalizing gold and predicted entities, such that “Hillary Clinton” and “Hillary Rodham Clinton” are unified, improved F1 to 57.3 F1. Extrapolating this to the test set would result in an F1 of 34.4