Relationship-Embedded Representation Learning for Grounding Referring Expressions
TABLE V: Ablation studies on variants of spatial relation graph of our proposed CMRIN on RefCOCO, RefCOCO+ and RefCOCOg. The variant is denoted as design of edge type with condition of existence for edges. The numbers in parentheses following the “type”, “dis” and “num” represent the number of types of edges, the threshold of normalized distance and the maximum number of edges at each vertices respectively. The two best performing models are marked in red and blue. CMRIN consistently outperforms existing state-of-the-art models on all the three benchmark datasets.
['[EMPTY]', '[EMPTY]', 'RefCOCO val', 'RefCOCO testA', 'RefCOCO testB', 'RefCOCO+ val', 'RefCOCO+ testA', 'RefCOCO+ testB', 'RefCOCOg val', 'RefCOCOg test']
On one hand, we adopt three types of designs for edges, i.e, “type(11)”, “type(7)” and “soft”. the “soft” is a fine-grained version of “type(11)” (row 1) is slightly worse than that of it with “type(11)” (row 7), because the design of “type(7)” is coarse than the design of “type(11)”. The CMRIN with “soft” (row 2) and “type(11)” (row 3) have similar performance on RefCOCO and RefCOCOg datasets, but the performance of latter is better than that of the former on RefCOCO+ dataset, which indicates that “type(11)” is fine enough to capture spatial relationships. In addition, “type(11)” is more memory- and computation-efficient than “soft”. On the other hand, we evaluate different conditions for connecting between objects, i.e, “edge num”, “axis dis” and “center dis”. In particular, the “edge num(5)” constraints the maximum out-degrees of each vertices to 5 and a vertex is connected to its 5 nearest nodes based on the distances between their normalized center coordinates (i.e., center distances) whose center distance is smaller than the threshold. (row 6) has relative lower precision than that with other conditions (row 3, 4, 5 and 7), because the “center dis(0.7)” covers several redundant edges which introduces noisy information. The CMRIN with remaining conditions have similar performance, and their differences in average precision over the validation and testing sets on RefCOCO, RefCOCO+ and RefCOCOg datasets are within ±0.07%, ±0.27% and ±0.55%, respectively.