Fluency-Guided Cross-Lingual Image Captioning
Table 1. Datasets for sentence fluency estimation. The relatively low rate of fluency (less than 30%) in machine-translated sentences indicates the importance of fluency-guided learning for cross-lingual image captioning.
['[EMPTY]', '[BOLD] training', '[BOLD] validation', '[BOLD] test']
Setup. In order to train the four-way sentence fluency classifier, a number of paired bilingual sentences labeled as fluent / not fluent are a prerequisite. We aim to select a representative and diverse set of sentences for manual verification, meanwhile keeping the manual annotation affordable. To this end we sample at random 2k and 6k English sentences from Flickr8k The 8k sentences were automatically translated into the same amount of Chinese sentences by the Baidu translation API. Manual verification was performed by eight students (all native Chinese speakers) in our lab. In particular, each Chinese sentence was separately presented to two annotators, asking them to grade the sentence as fluent, not fluent, or difficult to tell. A sentence is considered fluent if it does not contain obvious grammatical errors and is in line with language habits of Chinese. Sentences receiving inconsistent grades or graded as difficult to tell were ignored. This resulted in 6,593 labeled sentences in total. The fact that less than 30% of the translated sentences are considered fluent indicates much room for further improvement for the current machine translation system. It also shows the necessity of fluency-guided learning when deriving cross-lingual image captioning models from machine-translated corpus.