CMU LiveMedQA at TREC 2017 LiveQA: A Consumer Health Question Answering System
Table 1: Evaluation results on LiveMedQA system together with median and max performance.
['System', 'Avg Score', 'Success@ 2', 'Success@ 3', 'Success@ 4', 'Precision@ 2', 'Precision@ 3', 'Precision@ 4']
During official evaluation, in total 104 questions are received by the system and results of 103 of them are successfully submitted to the track server. Individual answers are evaluated by a 4-level Likert scale: excellent (4), good (3), fair (2) and bad (1). System overall performance is measured by 3 approaches: 1) the average score of all answers generated; 2) Success@i, which is the percentage of answers being scored as i among all questions received; and 3) Precision@i, which is the percentage of answers being scored as i among all answers generated. We find LiveMedQA system generally performs worse than average, and receives scores below median.