Learning Speech Representations from Raw Audioby Joint Audiovisual Self-Supervision
Table 6: The number of data samples in each split of each dataset.
['Dataset - % labels', 'Split size (samples) Train', 'Split size (samples) Val', 'Split size (samples) Test']
The concept of self-supervision is especially relevant to situations where labeled data is scarce. To compare the methods in such situations, we perform the same word classification experiments on the SPC and LRW datasets but with only 10% of the samples being used in the training set (the validation and test sets remain unchanged). Note that we completely omit the remaining 90% of the training set This leaves us with around 170 training examples per class for the SPC dataset (30 classes) and only around 20 training examples per class for the LRW dataset (500 classes). This makes the problem significantly more challenging. On SPC, there is a slight degradation in the performance of all methods. Our method attains an accuracy of 90.63% which is second to only wav2vec at an accuracy of 91.57%. On LRW, all other methods get severely affected and overfit to the small training set. Our method is the least affected and significantly outperforms all other methods with a best performance of 77.13%.