Relationship-Embedded Representation Learning for Grounding Referring Expressions
TABLE I: Comparison with the state-of-the-art methods on RefCOCO, RefCOCO+ and RefCOCOg. The two best performing methods using VGG-16 are marked in red and blue. The best performing method using ResNet is marked in bold.
['[EMPTY]', '[EMPTY]', 'feature', 'RefCOCO val', 'RefCOCO testA', 'RefCOCO testB', 'RefCOCO+ val', 'RefCOCO+ testA', 'RefCOCO+ testB', 'RefCOCOg val', 'RefCOCOg test']
Our proposed CMRIN consistently outperforms existing methods across all the datasets by a large margin, which indicates that our CMRIN performs well on datasets with different characteristics. Specifically, CMRIN improves the average Precision@1 over validation and testing sets achieved by the existing best-performing algorithm by 1.80%, 5.17% and 3.14% respectively on the RefCOCO, RefCOCO+ and RefCOCOg datasets when VGG-16 is used as the backbone network. Our CMRIN significantly improves the precision in the person category (test A of RefCOCO and RefCOCO+), which indicates that casting appearance attributes (e.g. shirt, glasses and shoes) of a person as external relationships between person and appearance attributes can effectively distinguish the target person from other persons. After we switch to the visual features extracted by ResNet-101 based Faster R-CNN, the Precision@1 of our CMRIN is further improved by another ∼4.40%. Note that our CMRIN only uses the 2048-dimensional features from pool5 while MattNet uses the feature maps generated from the last convolutional layers of both the third and fourth stages.