ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
Table 4: Cloze Test for ERNIE-ViL. Baseline model denotes pre-training without Scene Graph Prediction Tasks.
['[EMPTY]', 'objects R@1', 'objects R@5', 'attributes R@1', 'attributes R@5', 'relationships R@1', 'relationships R@5', 'overall R@1', 'overall R@5']
In the cloze test, language tokens represent detailed semantics (objects, attributes and relationships) are masked from the text and the model is required to infer them with the context from both text and image. To build the dataset, we sampled 15,000 image-text pairs from Flickr30K dataset and in total each of 5,000 object tokens, attributes and relationships are selected. And for the prediction, acc@1 and acc@5 are adopted as the evalation metric. An absolute improvement acc@1 of 2.12% for objects, 1.31% for relationships and 6.00% for attributes demonstrates that ERNIE-ViL learned better alignments for detailed semantics across modalities.