Fast Prototyping a Dialogue Comprehension System for Nurse-Patient Conversations on Symptom Monitoring
Table 2: QA model evaluation results. Each sample is a simulated multi-turn conversation.
['[BOLD] Training Samples', '[BOLD] 10k', '[BOLD] 50k', '[BOLD] 100k', '[BOLD] 150k']
To distinguish the correct answer span from the plausible ones which contain the same words, we measure the scores on the position indices of tokens. Our results show that both EM and F1 score increase with training sample size growing and the optimal size in our setting is 100k. The best-trained model performs well on both the Base Set and the Augmented Set, indicating that out-of-distribution symptoms do not affect the comprehension of existing symptoms and outputs reasonable answers for both in- and out-of-distribution symptoms. On the Real-World Set, we obtained 78.23 EM score and 80.18 F1 score respectively.