Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
Table 6: The number of codebook vectors at a particular VQ layer that learned to be a detector for any word with an F1 score greater than 0.5.
['# Quantized Layers', 'Model ID', 'VQ Layer', '# Word Detectors (F1 > 0.5)']
This suggests implicit word recognition ability can be localized at different layers in the ResDAVEnet audio model, and exactly where it emerges depends upon the VQ training curriculum. We hypothesize that this is due in part to two factors: In a warm-start model, whatever type of information (subword-like, word-like) the continuous model learned to encode at a particular convolutional layer (or residual block) does not change after a quantizer is appended to that layer. In a cold-start model, each active quantization layer forms a potential bottleneck, restricting the amount of information that is able to pass through to subsequent layers. According to this hypothesis, if word-level recognition tends to emerge at a particular layer in an unconstrained network with no quantization bottleneck, it will stay there when quantization is introduced for fine-tuning. However, when a quantization bottleneck is introduced from the very beginning of training, the gradient flowing down into the lower network layers is more constrained during the initial training epochs (when the gradient tends to be the largest). This may have the effect of steering the optimizer into a different part of the parameter space, in which word recognition occurs at a different layer than it otherwise would. Here, we claim that a codebook vector belonging to a particular VQ layer has learned to be a word detector if its F1 score for any word appearing in the test set exceeds 0.5 (as measured on the test set). There are several interesting things to note here. First, we only observe a significant number of word detectors at the VQ3 layer when that layer is trained from a cold-start. Even when adding a second VQ layer as in the {3}→{2,3} model, these detectors remain at VQ3.