Curate and Generate: A Corpus and Method for Joint Control of Semantics and Style in Neural NLG
Table 6: Automatic semantic evaluation (higher is better for all but SER).
['1', 'BLEU', '↑', 'Base 0.126', '+Adj 0.164', '+Sent 0.166', '+Style  [BOLD] 0.173']
Machine Translation Metrics. We begin with an automatic evaluation using standard metrics frequently used for machine translation. (weighted n-gram precision), where higher numbers indicate better overlap (shown with the ↑). We note that while these measures are common for machine translation, they are not well-suited to this task, since they are based on n-gram overlap which is not a constraint within the model; we include them for comparative purposes. This means that the base outputs errors are computed over only value slots (since they don’t contain adjectives), and the rest of the errors are computed over both value and adjective slots. Naturally, base, with no access to style information, has the best (lowest) SER. But we note that there is not a large increase in SER as more information is added – even for the most difficult setting, +style, the models make an error on less than 10% of the slots in a given mr, on average.