“President Vows to Cut Taxes Hair”:Dataset and Analysis of Creative Text Editing for Humorous Headlines
Table 4: LSTM accuracy for distinct grade bins (upper-bounds are inclusive) on the test set for X=40.
['Bin', '0-.4', '.4-.8', '.8-1.2', '1.2-1.6', '1.6-2.0', '2.0-2.4', '2.4-3.0']
The LSTM is able to distinguish the far extremes (≤0.4 and >1.6) of the test set much more convincingly than the headlines with mean grades in the interval (0.4,1.6]. We found a slightly negative correlation between classification accuracy and standard deviation of grades. Using additional judges for headlines with high standard deviation of grades would possibly improve annotator agreement and classification accuracy. The LSTM achieved a significantly lower accuracy when an entity (61.8%) was replaced by the micro-edit compared to when a noun (64.5%) or a verb (65.5%) was replaced. Although the classifier is never shown what has been replaced, it is better at assessing humor when the replaced word is not an entity. Our judges did have access to the replaced word, so we speculate this knowledge is important when the replaced word is an entity, especially when the entity triggers the judge’s recollection of their world knowledge surrounding the entity, which the LSTM does not have. Another potential reason is that the pretrained GloVe vectors are trained on web data (840 billion tokens obtained from Common Crawl) no more recent than 2014, which may not appropriately represent common entities in our 2017-2018 headline data.