Evaluating Creative Language Generation: The Case of Rap Lyric Ghostwriting
Table 3: The results of the automated evaluation. The bold indicates the system with a lower similarity at the target rhyme density.
['[BOLD] Artist', '[BOLD] Avg Rhyme Density', '[BOLD] Baseline  [BOLD] Similarity', '[BOLD] Baseline  [BOLD] N-gram', '[BOLD] LSTM  [BOLD] Similarity', '[BOLD] LSTM  [BOLD] iteration']
For each artist, we calculate their average rhyme density across all verses. We then use this value to determine at which iteration this rhyme density is achieved during generation (using the regression line for rhyme density). Next, we use the maximum similarity regression line to determine the maximum similarity score at that iteration. Low maximum similarity score indicates that we have maintained stylistic similarity while producing new, previously unseen lyrics. The reason is that in the beginning of training (in the LSTM’s case) and at a low n-gram length (for the baseline model), the models actually achieved a rhyme density that exceeded the artist’s average rhyme density. As a result, the rhyme density regression line hits the average rhyme density on a negative iteration.