Flexible and Creative Chinese Poetry Generation Using Neural Memory
Table 2: Preference ratios for systems with or without overfitting and with or without memory augmentation.
['[EMPTY]', 'Preference Ratio Compliance', 'Preference Ratio Fluency', 'Preference Ratio Theme', 'Preference Ratio Aesthetic', 'Preference Ratio Scenario']
Given a pair of poems generated by two different configurations using the same topic, the experts were asked to choose which one they preferred. The evaluation is therefore pair-wised, and each pair of configurations contains at least 180 evaluations. In particular, the ‘aesthetic innovation’ score for C∞ is not better than C1, which was different from what we expected. Informal offline discussions with the poetry experts found that the experts identified some innovative expression in the C∞ condition, but most of the them was regarded as being nonsense in the opinion of many of the experts. In comparison to sparking innovation, fluency and being meaningful is more important not only for non-expert readers, but also for professional poets. In other words, only meaningful innovation is regarded as innovation, and irrational innovation is simply treated as junk. For C1, the most substantial improvement is observed in terms of ‘Aesthetic innovation’, which is consistent with our argument that memory can help encourage innovation for this model. For C∞, ‘Fluency’ seems to be the most improved metric. This is also consistent with our argument that involving memory constrains over-innovation for over-fitted models. It seems that with the regularization, the overfitting problem is largely solved, and the generation is nearly as fluent and consistent as the C1 condition. Interestingly, the score for aesthetic innovation is also significantly improved. Since the regularization is not supposed to boost innovation, this seems confusing at first glance (in comparison to the result on the same metric in the first row), but this is probably because the increased fluency and consistency makes the innovation more appreciated, therefore doubly confirming our argument that true innovation should be reasonable and meaningful.