Context-Dependent Diffusion Network for Visual Relationship Detection
Table 4. Ablation Study (%) on VRD dataset.
['Feature', 'Predicate Det.', 'Predicate Det.', 'Relationship Det.', 'Relationship Det.', 'zero-shot Predicate Det.', 'zero-shot Predicate Det.', 'zero-shot Relationship Det.', 'zero-shot Relationship Det.']
Our proposed network uses the appearance and semantic cues. In this section, we will discuss their effects on the final performance. We first conduct an experiment without diffusion, i.e., not considering global context priors. Correspondingly, three tests are performed with appearance cue, semantic cue or both. We also report the predicate detection and relationship detection results on VRD dataset. It can be observed that semantic cue seems more robust than only appearance cue. The reason might be that semantic cue actually covers the categories and confidence scores of objects, which is the higher level features compared to appearance features. By combining both, the performance can be further improved by about 4% and 1% for predicate detection and relationship detection. It indicates that the two cues should be complementary for each other to some extend. Next, we test the influence of context priors. We can observe that integrating context information can further improve the performance. For semantic graphs, given a relationship triplet person-riding-elephant, we can borrow the information of relevant objects. For example, the cliques {person, people, girl, boy, man,⋯} and {elephant, horse, zebra, ⋯} should have high correlations in their internal connections. Thus, for the subject person and the object elephant in the triplet person-riding-elephant, we can get more shared information based on the graph connections. Intuitively, the shared information may be features of other objects. More implicitly, some relationships can be transferred into the current triplet. For example, we can infer new relationship triplets, such as girl-riding-elephant, boy-riding-horse, etc. By considering the current object information, the model can do a proper information aggregation for each object through graph diffusion. It indicates that the global context priors can enhance the generalization ability of visual relationship model to some extent. Further, by integrating two types of global context priors, the performance can be improved to 87.57% and 21.46% for the predicate and relationship detection respectively.