Mirroring to Build Trust in Digital Assistants
Table 2: F1-scores for the utterance chattiness and response chattiness preference binary classification tasks (chatty vs. non-chatty) for both the Leave One Participant Out (top) and the 80/20 evaluation splits (bottom).
['[BOLD] Leave One Participant Out', '[BOLD] Leave One Participant Out  [ITALIC] Log.', '[BOLD] Leave One Participant Out  [ITALIC] Naive', '[BOLD] Leave One Participant Out  [ITALIC] ANN', '[BOLD] Leave One Participant Out  [ITALIC] RF', '[BOLD] Leave One Participant Out  [ITALIC] RF', '[BOLD] Leave One Participant Out  [ITALIC] SVM', '[BOLD] Leave One Participant Out  [ITALIC] SVM', '[BOLD] Leave One Participant Out  [ITALIC] SVM', '[BOLD] Leave One Participant Out  [ITALIC] SVM', '[BOLD] Leave One Participant Out  [ITALIC] SVM']
This is a promising indicator that both a speaker’s degree of chattiness and their preference for chatty vs. non-chatty response can be detected reliably. The majority of classifiers had performance comparable to one another and better than chance, with the SVM methods performing best on both types of evaluation split and both classification tasks. These results are encouraging, especially given the small size of the data set. Performance on the 80/20 split indicates that the algorithms do not need many samples from a user to learn. Performance on the leave-one-participant-out split suggests that the models are able to generalize to new speakers. However, access to samples across participants does improve performance for all the classifiers tested.