Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval
Table 3. Question Relevance Prediction Performance. Best performances are marked in bold.
['[EMPTY]', 'Prec.', 'Recall', 'F1']
Relevant utterance prediction performance comparison. We can clearly observe that our BERT-CUR outperforms all the other prediction models. Although, the SVM-AWE model achieves the same Precision as BERT-CUR BASE, our proposed model supersedes the SVM-AWE in terms of Recall and hence F1-Measure as well. It states that in the majority of cases, the first turn in the conversation identifies the main topic of the conversation and the rest of the turns explore more details of the same topic, rather than shifting to a different topic. Notice that FirstUtterance is also the best performing heuristic for utterance prediction. Also, we observe that BERT-CUR LARGE also outperforms the baseline models. However, unlike other tasks, we observe better performance achieved by fine-tuning the BERT base model. This is because of smaller data size compared to the tasks where the BERT large model performs better.