Hal: Improved Text-Image Matching by Mitigating Visual Semantic Hubs
Table 2: Quantitative results on MS-COCO [23]. First three blocks (line 2.1-2.12) are using protocol c2 (5k test set); the last two blocks (line 2.13-2.24) is using c1 (1k test set) in convenience of comparing with results reported in previous works. Mb means memory bank.
['#', '[BOLD] architecture', '[BOLD] loss', '[BOLD] image→text R@1', '[BOLD] image→text R@5', '[BOLD] image→text R@10', '[BOLD] image→text Med r', '[BOLD] image→text Mean r', '[BOLD] text→image R@1', '[BOLD] text→image R@5', '[BOLD] text→image R@10', '[BOLD] text→image Med r', '[BOLD] text→image Mean r', 'rsum']
Comparing Hal, Sum and Max. On Flickr30k, we experiment three models and Hal achieves significantly better performance than Max and Sum on the first two configurations. Interestingly, while Max fails badly on Flickr30k, it becomes very competitive on MS-COCO. This serves as an evidence of Max easily overfitting to small datasets. In conclusion, Hal maintains its edge over Max and Sum across regardless of data and architecture configurations. Even without global weighting (memory bank), Hal still beats the two triplet losses by a large margin. The equipment of memory bank can usually further boosts rsum by another 3−5. Also, it is worth noticing that Hal converges significantly faster than Max and Sum. Hal stabilizes after approximately 5 epochs while Max and Sum take roughly 10 epochs. Hal vs. State-of-the-art. Notice that we are comparing against works that use frozen image encoder (as we do). With such prior, it is a well-established state-of-the-art on the Text-Image matching task, having much higher rsums than previous works. For SCAN, we pick configurations with the best rsums on both MS-COCO and Flikr30k, switching its learning objective from Max to Hal. On Flikr30k, Max and Hal deliver comparable results. On MS-COCO, Hal is significantly stronger - rsum is further improved to 512.7 with R@1 improved by 7.4 and 3.7 for image→text and text→image respectively. We did not experiment with Mb due to GPU memory limits.