VideoSET: Video Summary Evaluation through Text
Table 3: Agreement of VideoSET with human judgment, when choosing which of a pair of different subshots is semantically closest to a reference subshot. The comparisons are separated according to the VideoSET judgment, and the % of all cases for which the judgment occurs is listed. For each type of judgment, the % correct of VideoSET with respect to human judgment for 300 sampled comparisons is given, as well as the % correct of a pixel-based distance metric. Agreement using only non-zero human judgments in addition to all human judgments is given, since the large majority of human judgments evaluate both subshots in a pair to have zero similarity with the reference subshot. PB stands for pixel-based distance metric.
['VideoSET judgment', '% of cases', '% Correct (Using all human judgments) VideoSET', '% Correct (Using all human judgments) Pixel-based', '% Correct (Using non-zero human judgments) VideoSET', '% Correct (Using non-zero human judgments) Pixel-based']
At a finer level, we then assessed the performance of VideoSET on comparing pairs of individual subshots. Since the space is now more constrained, we densely computed VideoSET scores for every pair of subshots in the video with respect to every possible third subshot as a reference. We also computed scores based on the pixel-based distance metric. Based on these, we separated the comparisons into 4 different cases: (1) VideoSET judged both subshots to have no semantic similarity with the reference subshot; (2) VideoSET judged both subshots to have equal, non-zero semantic similarity with the reference subshot; (3) VideoSET judged one subshot to be semantically more similar than the other, and agreed with the pixel-based (PB) judgment; and (4) VideoSET judged one subshot to be semantically more similar than the other, and disagreed with the pixel-based (PB) judgment. We then sampled 300 comparisons from each of these 4 cases (a total of 1200 comparisons). For these samples, we asked humans to judge which subshot in each pair is semantically more similar to the reference subshot, if the pair is equally similar, or if both subshots have no similarity.