A Case Study on Combining ASR and Visual Featuresfor Generating Instructional Video Captions
(b) Hardest (AUCμ,w)
['4', '43.8']
Specifically — for each word type w (e.g., w=beer) we measure how well w is predicted by the classifier based on ASR / spoken tokens AUCt,w (e.g., AUCt,beer = 98) and, conversely, how well w is predicted by the visual classifier AUCv,w (AUCv,beer = 68). For a given word type, we measure its overall difficulty by averaging AUCt,w and AUCv,w; we call this AUCμ,w (AUCμ,beer = 83). Similarly, we measure the difference in difficulty by subtracting AUCt,w and AUCv,w to give AUCΔ,w (AUCΔ,beer = 30) with higher values indicating that a word type is predicted better by the spoken-token features compared to the visual features. We plot AUCt,w versus AUCv,w for 382 words in Fig. Absolute Performance. Points in the upper-right quadrant of Fig. Specific ingredients, e.g., “nori” and “mozzarella,” are often easy to detect, as are actions closely associated with particular objects (e.g., “dough” is almost always the object being “knead”-ed). Conversely, pronouns (e.g., “it”) and conjunctions (e.g., “or”) are universally difficult to predict. Visual vs. ASR-token–based features. In general, ASR-token–based features carry greater predictive power, as evidenced by the skew towards the bottom right in the scatterplot in Fig. One pattern in the cases where speech features perform better Indeed, small, detailed distinctions may be often difficult to make from visual features, e.g., “vegetable oil” and “olive oil” may look identical in most YouTube videos.