The Web as a Knowledge-base for Answering Complex Questions
Table 6: F1 results for ComplexQuestions.
['[BOLD] System', '[BOLD] Dev. F1', '[BOLD] Test F1']
By analyzing the training data, we found that we can decompose Comp questions with a rule that splits the question when the words “when” or “during” appear, e.g., “Who was vice president when JFK was president?”. We decomposed questions with this rule and obtained 39.7 F1 (SplitQARule). Analyzing the development set errors, we found that occasionally SplitQARule returns a correct answer that fails to string-match with the gold answer. By manually fixing these cases, our development set F1 reaches 46.9 (SplitQARule++). Note that CompQ does not suffer from any string matching issue, as it operates directly against the Freebase KB and thus is guaranteed to output the answer in the correct form. This short experiment shows that a web-based QA model can rival a semantic parser that works against a KB, and that simple question decomposition is beneficial and leads to results comparable to state-of-the-art.