Curate and Generate: A Corpus and Method for Joint Control of Semantics and Style in Neural NLG
Table 7: Automatic stylistic evaluation metrics (higher is better). Paired t-test base vs. +style all p<0.05.
['1', 'Vocab', '↑', 'Base  [BOLD] 8,627', '+Adj 8,283', '+Sent 8,303', '+Style 7,878']
For vocab, we find the number of unique words in all outputs for each model. We find the average sentence length (SentLen) by counting the number of words, and find the total number of times an adjective is used (Row 3) and average number of adjectives per reference for each model (Row 4). We compute Shannon text entropy (E) as: E=−∑x∈Vft∗log2(ft), where V is the vocab size in all outputs generated by the model, f is the frequency of a term (in this case, a trigram), and t counts the number of terms in all outputs. Finally, we count the instances of contrast (e.g. “but” and “although”), and aggregation (e.g. “both” and “also”). For all metrics, higher scores indicate more variability (indicated by ↑).