Learning Speech Representations from Raw Audioby Joint Audiovisual Self-Supervision
Table 1: Results for spoken word classification (Accuracy in %) on the Speech Commands (SPC, 30 classes) (Warden, 2018) and the Lip Reading in the Wild (LRW, 500 classes) (Chung and Zisserman, 2016) datasets. For evaluation, a 2 layer GRU model is used on the encoder outputs for each pretraining method, before finetuning on the downstream task.
['Pretraining method', 'Self-supervision', 'Input type', 'Dataset and % of Labels used SPC', 'Dataset and % of Labels used SPC', 'Dataset and % of Labels used LRW', 'Dataset and % of Labels used LRW']
We evaluate all methods on isolated word classification on the Speech Commands (SPC) We use a 2 layer BiGRU (with 256 units in each layer) on the encoder outputs followed by a linear layer with as many units as the number of target classes (30 for SPC, 500 for LRW). This acts as the downstream classifier and remains the same for every method. For downstream classifiction, we finetune the models (as shown in bottom of Fig. The learning rate is 0.0001 for the first 40 epochs and 0.00001 for the last 10 epochs. We use the standard softmax + cross entropy loss for training. We opted to use a BiGRU for simplicity, however this can be replaced by any model that can classify variable length sequences into discrete categories (such as LSTMs, TCNs, LiGRUs .