On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval
None
['Hours', 'Textual baseline', 'Visual baseline', 'MTL ensemble']
For our MTL models, we can use either output (MTL-visSup or MTL-textSup) or the average of the two (ensemble). Note that the visual baseline result is a horizontal line in each plot, as this baseline does not use any textual supervision. We present the results for Spearman’s ρ, which evaluates with respect to the “soft” human labels, and average precision, which uses the hard majority decisions. The trends for other hard label-based metrics (P@10, P@N) follow the same trend as average precision. Effect of representation loss. Removing the latter reduces average precision by roughly 2-4% for most points on the curve, while for Spearman’s ρ it makes little difference. However, when tuning for exact retrieval on the development set, we find much larger gains from the representation loss, roughly 7-15% in average precision. This difference could be an artifact of tuning and testing on different tasks.