Ultrasound tongue imaging for diarization and alignmentof child speech therapy sessions
Table 2: Results for word alignment (precision, recall, F1) and recognition (WER) for typically developing speech (UXTD) and disordered speech (UXSSD). Baseline systems use GMMs trained on Ultrasuite and PF-Star data. USuite systems use neural network acoustic models. Highlighted results indicate best performing systems.
['[BOLD] Systems', '[BOLD] Pre', '[BOLD] Rec', '[BOLD] F1', '[BOLD] WER']
We train the acoustic models with the Kaldi speech recognition toolkit [povey2011kaldi]. GMMs are initialized on pooled data from UXTD (1.75 hours), UPX (11.05 hours), and the training set of the PF-STAR children’s speech corpus (7.4 hours) [batliner2005pf_star]. The inclusion of the PF-STAR corpus is beneficial for model initialization as it contains additional manually transcribed data for 86 speakers. This way, we avoid aligning target words with time segments belonging to the speech therapist. After monophone and triphone training, Mel Frequency Cepstral Coefficients (MFCCs) are processed with Linear Discriminant Analysis (LDA) and a Maximum Likelihood Linear Transform (MLLT). This is followed by Speaker Adaptive Training (SAT) with feature-space MLLR (fMLLR) [rath2013improved, fainberg2016improving]. Alignment and fMLLR features from this system are then used to train feedforward neural networks, following Kaldi’s nnet1 recipe. Diarization is not beneficial for the typically developing dataset, as it has a small amount of adult speech. This is not the case for disordered speech, where diarization leads to more accurate alignment and recognition. Although baseline GMMs perform reasonably well in terms of alignment, they do not outperform neural network acoustic models in terms of recognition. Augmenting in-domain data with ultrasound leads to improvements on both datasets, but not as much as using additional out-of-domain acoustic data. The best system is a combination of the systems augmented with out-of-domain data and ultrasound embeddings. Although we observe small improvements in terms of alignment metrics, WER shows that combined models are more robust than either of them separately. Best results occur when there is a slight bias towards the system augmented with out-of-domain training data. These results also show that ultrasound and audio data can complement each other well.