Team EP at TAC 2018: Automating data extraction in systematic reviews of environmental agents
Table 3: The estimation of impact of various design choices on the final result. The entries are sorted by the out-of-fold scores from CV. The SUBMISSION here uses score from ep_1 run for the single model and ep_2 for the ensemble performance.
['ID LSTM-800', '5-fold CV 70.56', 'Δ 0.66', 'Single model 67.54', 'Δ 0.78', 'Ensemble 67.65', 'Δ 0.30']
The results are presented in Table 3. [CONTINUE] Perhaps the most striking thing about the ablation results is that the 'traditional' LSTM layout outsperformed the 'alternating' one we chose for our submission. [CONTINUE] Apart of the flipped results of the LSTM-800 and the LSTM-400, small differences in CV score are sometimes associated with large discrepancies in test set performance. This is mostly due to small size of the data set (low [CONTINUE] Some ablated models that perform poorly in the singlemodel scenario (e.g. NO-OVERLAPS, LSTM-400DROPOUT) are able to regain a lot of accuracy when ensembled. [CONTINUE] Also, our data augmentation technique (NO-TRANSLATIONS) seem to have far smaller impact on the final score then we expected.