Towards Multimodal Understanding of Passenger-Vehicle Interactions in Autonomous Vehicles: Intent/Slot Recognition Utilizing Audio-Visual Data
Table 1: Speech Embeddings Experiments: Precision/Recall/F1-scores (%) of NLU Models
['[BOLD] Modalities', '[BOLD] Features (Embeddings)', '[BOLD] Intent Recognition Prec', '[BOLD] Intent Recognition Rec', '[BOLD] Intent Recognition F1', '[BOLD] Slot Filling Prec', '[BOLD] Slot Filling Rec', '[BOLD] Slot Filling F1']
When used in isolation, Word2Vec and Speech2Vec achieves comparable performances, which cannot reach GloVe performance. This was expected as the pre-trained Speech2Vec vectors have lower vocabulary coverage than GloVe. Yet, we observed that concatenating GloVe + Speech2Vec, and further GloVe + Word2Vec + Speech2Vec yields better NLU results: F1-score increased from 0.89 to 0.91 for intent recognition, from 0.96 to 0.97 for slot filling.