Coherent Multi-Sentence Video Description with Variable Level of Detail
Table 3: Visual recognition of SR, accuracy in %.
['Approach', 'acti.', 'tool', 'obj.', 'source', 'target', 'dish']
We first evaluate the output of our visual recognition, the SR. We notice that the recognition of the handled object (in many cases the ingredient), is the most difficult, achieving only 33.2% compared to 60.8% or more for the other nodes. This is in contrast to the importance for verbalization where the activity (second lowest) and handled object are naturally most important (see also Sec. However, the dish recognition of 8.1% is too low and enforcing consistency by conditioning on the node prediction confuses the other nodes, leading to a drop in performance for most nodes. Next we improve the dish recognition accuracy by adding more training data during the CRF training. that correspond to dishes of the TACoS subset. This data was previously only used for learning attribute/unary classifiers. This leads to an improvement not only for the dish node, but for all nodes (see line 4). As a next step we add unaries to the dish node. During training and test time we use the ground-truth segmentation for computing the features. Comparing this to the same approach without dish features (line 5 versus 4) improves the dish node significantly from 29% to 46%. As a last step we add our hand centric color Sift features as second unary for all states from the nodes tool, object, source, and target. This leads to a significant improvement for objects of 6.7% and for dish of 9.9% (line 6 versus 5).