Speech-Based Visual Question Answering
Table 1. Dimensions for the conv layers. Example shown with a 2 second long audio waveform, sampled at 16 kHz. The final output dimensions are (3, 512)
['Layer', 'conv1', 'pool1', 'conv2', 'pool2', 'conv3', 'pool3', 'conv4', 'pool4', 'conv5']
Convolution layers are used to encode waveforms because they reduce dimensionality of data while finding salient patterns. The maximum length of a spoken question in our dataset is 6.7 seconds and corresponds to a waveform length of 107,360 elements, while the minimum is 0.63 seconds and corresponds to 10,080 elements. One could directly feed the input waveform to a LSTM, but a LSTM will be unable to learn from sequences that are excessively long, so dimensionality reduction is a necessity. Each consecutive convolution layer halves in filter length but doubles the number of filters. This is done for simplicity rather than for performance optimization. The main consideration taken in choosing the parameters is that the last convolution should output dimensions of (x, 512), where x must be a positive integer. x represents the length of a sequence of 512-dim vectors. The sequence is then fed into an LSTM, which then outputs a single vector of (512). Thus, x should not be too big, and the CNN parameters are chosen to ensure a sensible sequence length. The longest and shortest waveforms correspond to final convolution outputs of size (13, 512) and (1, 512) respectively. 512 is used as the dimension of the LSTM to be consistent with TextMod and the original VQA baseline.