Imagine This! Scripts to Compositions to Videos
Table 5: Human evaluation to estimate consistency and quality of generated videos.
['[EMPTY]', '[BOLD] Composition Consistency  [BOLD] Position', '[BOLD] Composition Consistency  [BOLD] Rel. Size', '[BOLD] Composition Consistency  [BOLD] Interact.', '[BOLD] Visual Quality  [BOLD] FG', '[BOLD] Visual Quality  [BOLD] BG', '[BOLD] Visual Quality  [BOLD] Sharpness']
Modelling Pixels vs Retrieval. We experimented extensively with text conditioned whole video generation using models with and without adversarial losses and obtained poor results. Since generative models tend to work better on images with single entities, we swapped out the target embedding network in the entity retriever by a generator. Given the query embedding at each of the F time steps, the generator produces an appearance image and a segmentation mask. The model is trained using an L1 loss between the masked appearance image and the masked ground truth image, and an L1 loss between the generated and ground truth masks. See supplementary material for more details. This baseline produced blurry results with recognizable colors and shapes for most common characters like Fred, Wilma, Barney, and Betty at best. We also tried GAN and VAE based approaches and got only slightly less blur. Moreover, since the visual quality of the generated previous entities affects the performance of the layout composer, this also translates into poor ratings on the composition consistency metric. Since the semantic fidelity metrics can not be computed for this pixel generation approach, we ran a human evaluation to compare this model to ours. Humans were asked to mark nouns, adjectives and verbs in the sentence missing in the generated video. Craft significantly outperformed the pixel generation approach on noun, adjective, and verb recall (Craft 61.0,54.5,67.8, L1: 37.8,45.9,48.1).