Localizing Moments in Video with Temporal Language
Table 7: Comparison of different methods to localize context fragments (e.g., the text “she bends down” in the sentence “the girl talks after she bends down”). We compare localizing fragments with the MLLC model to localizing fragments with the latent context considered when localizing the whole query.
['[EMPTY]', 'Before R@1', 'Before mIoU', 'After R@1', 'After mIoU']
Localizing Context Fragments. TEMPO-HL sentences can be broken into two parts: a base-sentence fragment (which refers to the base moment), and a context-sentence fragment (which refers to the context moment). For example, for the sentence “The girl holds the ball before throwing it,”, “the girl holds the ball” is the base fragment and “throwing it” is the context fragment. A majority of the “before” and “after” sentences in TEMPO-HL are of the form “X before (or after) Y”, so we can determine a list of sentence fragments by splitting sentences based on the temporal word. Given “before” and “after” sentences, we determine the ground truth context fragment by considering which reference moment was given to annotators. We can then measure how well models localize context fragments. We find that our model reliably selects the correct context fragments, most likely because it can properly exploit temporal understanding of how the context fragment relates to the base fragment.