Key-Value Memory Networks for Directly Reading Documents
Table 5: Analysis of test set results (% hits@1) for KB vs. Synthetic Docs on WikiMovies.
['Knowledge Representation', 'KV-MemNN']
For a given movie, we use a simple grammar to construct a synthetic “Wikipedia” document based on the KB triples: for each relation type we have a set of template phrases (100 in total) used to generate the fact, e.g. “Blade Runner came out in 1982” for the entry Blade Runner release_year 1982. We can then parameterize the complexity of our synthetic documents: (i) using one template, or all of them; (ii) using conjunctions to combine facts into single sentences or not; and (iii) using coreference between sentences where we replace the movie name with “it”. The purpose of this experiment is to find which aspects are responsible for the gap in performance to a KB. They indicate that some of the loss (93.9% for KB to 82.9% for One Template Sentence) in performance is due directly to representing in sentence form, making the subject, relation and object harder to extract. Moving to a larger number of templates does not deteriorate performance much (80%). The remaining performance drop seems to be split roughly equally between conjunctions (74%) and coreference (76%). The hardest synthetic dataset combines these (All Templates + Conj. + Coref.) and is actually harder than using the real Wikipedia documents (72.5% vs. 76.2%). This is possibly because the amount of conjunctions and coreferences we make are artificially too high (50% and 80% of the time, respectively).