Open Question Answering with Weakly Supervised Embedding Models
Table 6: Performance of our embedding model for retrieving answers for questions from the WebQuestions test set, among the whole ReVerb KB (14M candidates).
['[BOLD] Method', '[EMPTY]', '[BOLD] Top-1', '[BOLD] Top-10', '[BOLD] F1']
For each test question, we record the rank of the first ReVerb triple containing the answer entity. Top-1 and Top-10 are computed on questions for which the system returned at least one answer (around 1,000 questions using string matching), while F1 is computed for all questions. One of the main reasons is that most questions of WebQuestions, such as Who was vice-president after Kennedy died?, should be represented by multiple triples, a setting for which our system has not been designed. Still, for a system trained with almost no manual annotation nor prior information on another dataset, with an other –very noisy– KB, the results can be seen as particularly promising. Hence, there might be higher ranked answers but they are missed by our evaluation script.