Identifying Visible Actions in Lifestyle Vlogs
Table 6: Results from baselines and our best multimodal method on validation and test data. ActionG indicates action representation using GloVe embedding, and ActionE indicates action representation using ELMo embedding. ContextS indicates sentence-level context, and ContextA indicates action-level context.
['Method', 'Input', 'Accuracy', 'Precision', 'Recall', 'F1']
Concreteness. We label as visible all the actions that have a concreteness score above a certain threshold, and label as non-visible the remaining ones. We fine tune the threshold on our validation set; for fine tuning, we consider threshold values between 3 and 5. Feature-based Classifier. For our second set of baselines, we run a classifier on subsets of all of our features. We run experiments with various combinations of features: action GloVe embeddings; POS embeddings; embeddings of sentence-level context (ContextS) and action-level context (ContextA); concreteness score. We build a similar model that embeds actions using ELMo (composed of 2 bi-LSTMs). We pass these embeddings through the same feed forward network and sigmoid activation function. Yolo Object Detection. Our final baseline leverages video information from the YOLO9000 object detector. This baseline builds on the intuition that many visible actions involve visible objects. We thus label an action as visible if it contains at least one noun similar to objects detected in its corresponding miniclip. To measure similarity, we compute both the Wu-Palmer (WUP) path-length-based semantic similarity Wu and Palmer For every action in a miniclip, each noun is compared to all detected objects and assigned a similarity score. As in our concreteness baseline, the action is assigned the highest score of its corresponding nouns. We use the validation data to fine tune the similarity threshold that decides if an action is visible or not. The model that uses all the input features available leads to the best results, improving significantly over the text-only and video-only methods.