Neural Wikipedian: Generating Textual Summaries from Knowledge Base Triples
Table 6: Automatic evaluation with the perplexity (i.e. lower is better), BLEU and ROUGEL metric (i.e. higher is better) on the validation and the test set. The average performance of the baseline along with its standard deviation is reported after sampling 10 times.
['[BOLD] Model', '[BOLD] Perplexity Valid.', '[BOLD] Perplexity Test', '[BOLD] BLEU 1 Valid.', '[BOLD] BLEU 1 Test', '[BOLD] BLEU 2 Valid.', '[BOLD] BLEU 2 Test', '[BOLD] BLEU 3 Valid.', '[BOLD] BLEU 3 Test', '[BOLD] BLEU 4 Valid.', '[BOLD] BLEU 4 Test', '[BOLD] ROUGEL Valid.', '[BOLD] ROUGEL Test']
To demonstrate the effectiveness of our system, we compare it to two baselines. First, we compute expected lower bounds for BLEU scores by using a random Wikipedia summary generation baseline. We regard the latter as a particularly strong baseline, due to the fact that Wikipedia biographies tend to follow a limited number of textual patterns. For each triples set on the validation and test set, the random system generates a response by randomly selecting a Wikipedia summary from our training set. Secondly, we use the KenLM toolkit Heafield et al. in order to build a 5-gram Kneser-Ney (KN) language model. During testing, similarly to the case of our neural network approach, for each triple set in the validation and test set, we use beam-search with a beam of size 10, in order to generate the 10 most probable summaries. We equip both of our baselines with surface form tuples, and the   and property-type placeholders. After a summary is selected, its   placeholder along with any potential property-type placeholders are replaced based on the original triples.