Learning Articulated Motion Models from Visual and Lingual Signals
TABLE II: Detailed success and failure analysis
['[EMPTY]', '[EMPTY]', 'Manual Dict. Success', 'Manual Dict. Ambig.', 'Word Embedding Success', 'Word Embedding Ambig.', 'Word Embedding WA']
In order to better understand the effects of variability in the linguistic input, we then asked the user to generate four additional diverse captions for each video. As can be expected, the model selection accuracy is greater for some captions when using the oracle dictionary. We attribute this difference to two primary factors. First, some captions describe the motion of parts using words that are ambiguous in their meaning. For example, several captions include the term “pull,” which may refer to both prismatic or rotational motion according to both the manual dictionary and the word embedding representation, i.e., dist(``pull'',Wpri)≃dist(``% pull'',Wrot), where Wpri,Wrot are the vectors that represent the two kinematic types. Second, the word embedding-based method may yield inaccurate estimates of word similarity as a result of having been trained on general-domain text. For example, while “slide” is only in the manually defined dictionary for prismatic motion, the word embeddings suggest that it is equidistant from the centroid for each of these types, i.e., dist(``slide'',Wpri)≃dist(``% slide'',Wrot). Note that ambiguities that result from similarity in the word embedding space distances are different from ambiguities inherent in the verb itself. The fraction that fail due to the ambiguity inherent in the specific verb itself is denoted as “Ambig.” and the fraction that are successful is represented as “Success.” The total number of description-video pairs is calculated based up upon five descriptions per video. Note that the multimodal nature of our model allows the visual signal to mitigate ambiguity in the lingual observation. In this way, it is possible to use visual cues to overcome failures of the linguistic models just as we use the lingual signal to mitigate failure of the visual pipeline.