Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech
Table 4: Left: Maximum R@1 (in %) on a speech-image retrieval task on Flickr8k using two \textscGRU\textscPACK.-- layers (left). Right: Summary of the results shown in Table 3 for phones and words for comparison purposes.
['[BOLD] Hierarchical  \\textsc  [ITALIC] GRU \\textsc  [ITALIC] PACK.--', '[BOLD] Phones Words', '\\textsc  [ITALIC] GRU \\textsc  [ITALIC] PACK.--', '[BOLD] Phones', '[BOLD] Words']
In this work we explore hierarchical segmentation with phones and words on the Flickr8k data set, and leave syllables for future work. We only consider the keep condition as it yields better results than the all condition. We vary the position of the \textscGRU\textscPACK.-- layers and test all positions where two \textscGRU\textscPACK.-- layers follow one another. For each configuration, the lowest \textscGRU\textscPACK.-- will receive phone boundary information, and the next layer will receive word boundary information. Note that such configuration results in a double sequence reduction. Indeed, after the first \textscGRU\textscPACK.-- layer, they are only as many output vectors as they are phones, and in the second, the resulting phone vectors are recomposed together to form words, resulting in as many output vectors as they are words.