Unsupervised Word Segmentation from Speech with Attention
Table 2: Precision, Recall and F-measure on word boundaries over the Mboshi5k corpus, using different AUD to extract pseudo-phones from speech. True phones toplines are also provided. *averaged scores over 5 different runs; +averaged 5 attention matrices
['[BOLD] AUD feat.', '[BOLD] AUD model', '[BOLD] dpseg (monoling.)  [BOLD] P', '[BOLD] dpseg (monoling.)  [BOLD] R', '[BOLD] dpseg (monoling.)  [BOLD] F', '[BOLD] proportional baseline (biling.)  [BOLD] P', '[BOLD] proportional baseline (biling.)  [BOLD] R', '[BOLD] proportional baseline (biling.)  [BOLD] F', '[BOLD] attentional (biling.)*  [BOLD] P', '[BOLD] attentional (biling.)*  [BOLD] R', '[BOLD] attentional (biling.)*  [BOLD] F', '[BOLD] att. average (biling.)+  [BOLD] P', '[BOLD] att. average (biling.)+  [BOLD] R', '[BOLD] att. average (biling.)+  [BOLD] F']
We trained 5 different NMT models changing the train/dev (biling.) The latter slightly boosts boundary detection performance. For all AUD configurations, our method outperforms two baselines (dpseg and proportional), as well as a pure speech-based baseline using segmental DTW Conversely, our method is much more robust to noise and seems better suited for real-world scenarios. While straightforward, the bilingual baseline (proportional) is rather strong compared to its monolingual counterpart (dpseg). This suggests that multilingual grounding provides a useful signal for word segmentation from speech. The NMT system requires a sequence of unsegmented symbols (the phones) and their aligned sentence translations in order to provide segmentation. Therefore, the AUD method chosen to encode the speech input has an impact on the quality of the final segmentation. Our best word segmentation results One natural explanation would be to posit that phone boundaries (and consequently word boundaries) are more accurately detected by the SVAE model than the HMM model. This improvement also holds for tokens and types scores (not reported here). However, while the topline achieves 34.3% of vocabulary (types) retrieval, our best AUD setup achieves 13.5% only. This illustrates the difficulty of the word discovery task – a task already challenging with true phones – in noisy setups. The large difference between true phones and pseudo-phones for type’s retrieval could be explained by the fact that a single change in the pseudo-phone sequence representing two speech segments of a same word will have the consequence to split the word cluster in two parts (in two types). A deeper analysis of the word clusters obtained is probably necessary to better understand how AUD from speech affects the word discovery task, and to come up with ways to better cluster speech segments in relevant types.