Eliciting Knowledge from Experts: Automatic Transcript Parsing for Cognitive Task Analysis
Table 1: Matching performance on the manual matching testset with different sentence encoding, in token level accuracy and mention level accuracy. BERT fea. means using features extracted by BERT model. and Exact is the exact matching baseline
['Encoding', 'Tok. Acc.', 'Tok. F1', 'Men. Acc.']
Evaluation. We evaluate the performance of our text spans matching methods with the manual matching test set by token level metrics and mention level accuracy, where token level metrics are normalized by sentence lengths. Glove-300d shows better token level accuracy and F1 score while BERT features have a better mention level accuracy. For cheaper computation, we use Glove-300d as the sentence encoding method of matching for the following sections. Please refer to the appendix for the case study of matching.