Better Text UnderstandingThrough Image-To-Text Transfer
Table 3: Results of applying different text encoders to the VETE model. The training data is MS COCO, and the RNN-based models learned to model this distribution better. However, BOW generalizes better to other datasets.
['[BOLD] Encoder', '[BOLD] images2014', '[BOLD] images2015', '[BOLD] COCO-Test', '[BOLD] Pin-Test']
We study the impact of the different text encoders on the VETE model. For BOW, we try two options: either we use the sum or the mean of word embeddings. Both bag-of-words encoders perform better than RNN encoders, although RNNs are slightly better on the test data which has the same distribution as the training data. Covariance: Cov(x,y). The Pearson correlation ρ: measures the linearity of the link between two variables, estimated on a sample; it is defined as ρ(x,y)=Cov(x,y)Std(x)Std(y). Surrogate Kendall τ: The Pearson correlation takes into account only linear dependencies. To mitigate this, we experimented with the Kendall correlation τ which is only rank-dependent. Unfortunately, it is not differentiable. We therefore used its differentiable approximation: Rank loss: Another cost function is the pairwise ranking loss.