Neural Naturalist: Generating Fine-Grained Image Comparisons
Table 2: Experimental results for comparative paragraph generation on the proposed dataset. For human captions, mean and standard deviation are given for a one-vs-rest scheme across twenty-five runs. We observed that CIDEr-D scores had little correlation with description quality. The Neural Naturalist model benefits from a strong joint encoding and Transformer-based comparative module, achieving the highest BLEU-4 and ROUGE-L scores.
['[EMPTY]', 'Dev BLEU-4', 'Dev ROUGE-L', 'Dev CIDEr-D', 'Test BLEU-4', 'Test ROUGE-L', 'Test CIDEr-D']
We observe improvement across BLEU-4 and ROUGE-L scores compared to baselines. Curiously, we observe that the CIDEr-D metric is susceptible to common patterns in the data; our model, when stopped at its highest CIDEr-D score, outputs a variant of, “these animals appear exactly the same” for 95% of paragraphs, nearly mimicking the behavior of the most frequent paragraph (Freq.) baseline. The corpus-level behavior of CIDEr-D gives these outputs a higher score. We observed anecdotally higher quality outputs correlated with ROUGE-L score, which we verify using a human evaluation (paragraph after next).