Coin_flipper at eHealth-KD Challenge 2019
Table 1: F1 scores, Precision and Recall obtained for the three scenarios achieved by baseline system, best method presented and our method.
['[EMPTY]', 'baseline F1', 'baseline P', 'baseline R', 'Joint-BERT-RCNN (best) F1', 'Joint-BERT-RCNN (best) P', 'Joint-BERT-RCNN (best) R', 'Voting LSTMs (ours) F1', 'Voting LSTMs (ours) P', 'Voting LSTMs (ours) R']
Our results were very competitive. We were second in the main evaluation (scenario 1) by a narrow margin - only 1.76 of F1 score over 100. This is surprising because scenario 1 is the composition of the other 2 scenarios. The main difference in evaluation among scenarios was that for scenario 1 a subset of 100 sentences out of a 8.800 sentences dataset was randomly selected for evaluation, while in scenarios 2 and 3 a dataset of 100 sentences was given to each one for evaluation. In subtask B, we noticed that not all relations had the same difficulty in being detected. We found that a first set of 6 relations (in-context, subject, target, domain, arg and is-a) were successfully identified with high recall and precision values when tested on development dataset (above 75%). Relations in-place, has-property, same-as and in-time were (in this order) harder to identify and, finally, for part-of, causes and entails relations, results were so bad that models for those relations were discarded.