MIDAS, A Dialog Act Annotation Scheme for Open-Domain Human-Machine Spoken Conversations
Table 2: BERT_F-DA+text achieves the best precision and F1 score. Results reported are an averaged score of six different random seed runs.
['[EMPTY]', 'Pre(%)', 'Rec(%)', 'F1(%)']
Another method to incorporate history is to replace the actual previous segment unit with its dialog act labels (if there are two labels for one segment, we combine both labels). Transformer models using BERT embeddings (BERT-text) outperform Bi-LSTM models with pre-trained word embeddings (LSTM-text) by a large margin (from 75.51% to 79.11% in F1). If we further fine tune the BERT language model on an unsupervised training task with similar data distribution (BERT_F-text), the classification result further improves from 79.11% to 79.40% in F1. This is consistent with previous research on in-domain pre-training Siddhant et al. However, the performance improvement is not statistically significant. One possible reason is that models pre-trained on a very large text dataset, such as Wikipedia, already encodes sufficient semantics for dialog act prediction. Therefore, fine-tuning the model on a more domain aligned data set does not improve the performance drastically.