Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets
Table 4: Performance of the APC model as a function of the dataset size.
['Percentage', 'across-speaker', 'within-speaker']
Unlike earlier, the model was trained with a learning rate of 10−5, as this was found to improve training stability in the earlier experiments. Considering the strong correlation between MAE and the ABX-scores, each model was chosen based on the lowest validation loss. The differences in the ABX-scores are relatively negligible when taking into account that the models were trained for a maximum of 100 epochs (usually with the lowest validation loss value). This implies that the models could still improve their representations with more training. That being said, with only 25% of the total data, that is 6.3 h of the French dataset, the APC model already converged with the hyperparameters here defined. Contradictory to the idea that more training data improves the performance, this result shows that hyperparameter tuning would be more beneficial in this case than increasing the training data. For CPC, we could not conduct the experiments in time.