Data Augmenting Contrastive Learning of Speech Representationsin the Time Domain
Table S1: Architecture ablations, ABX errors (Libri-light dev set). We compare the original CPC model described in [riviere2020multi] with modifications including more LSTM layers and a single Multi-Head prediction model for all time steps (MH). The bottom model is the one we refer to as CPC2 in the paper.
['[EMPTY]', 'Within spk. dev', 'Within spk. dev', 'Across spk. dev', 'Across spk. dev']
CPC2 is a modified version of the CPC architecture in [kahn2020, riviere2020multi]. The encoder architecture is unchanged (5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256). For the recurrent context nextwork, we use a 2-layer LSTM, as a tradeoff between feature quality and training speed. In the prediction network, we replace the k independent transformers in [kahn2020, riviere2020multi], each one predicting a specific time-step ahead, to a single multi-head transformer layer with k classifiers at its heads. This has a limited impact on accuracy but dramatically decreases training time. We started from the model described in [riviere2020multi]: the encoder network is composed of 5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256. We worked with ReLU activation and inserted a channel normalization procedure between each convolutional layer. As far as the context network is concerned, we used a 2-layers LSTM. Finally, we used a single layer multihead transformer to do the prediction instead of several single head transformers. We observe that in the cases of 3h and 45h datasets, the architecture with 2 layers of LSTM still perform best. However, with 100h of data, increasing the model depth turns out to be beneficial.