SRL4ORL: Improving Opinion Role Labeling UsingMulti-Task Learning With Semantic Role Labeling
Table 2: Datasets w/ nb. of SRL predicates/ORL opinions in train, dev & test set, size of label inventory.
['[EMPTY]', 'task', 'train size', 'dev size', 'test size', '|Y|']
SRL4ORL. The semantic roles of the predicate fear (marked blue bold) correspond to the opinion roles H and T, according to MPQA. We evaluate our models using two evaluation setting. However, in the 10-fold CV setting, the test sets are more than 3 times smaller than the dev set Therefore we additionally evaluate our models with 4-fold CV. We set aside 100 documents for development and use 25% of the remaining documents for testing. We run 4-fold CV twice with two different random seeds. All HPs can be found in the Appendix. The FS-MTL and the H-MTL models improve significantly in all evaluation measures, for both holders and targets, on both dev and test sets, when evaluated with repeated 4-fold CV. With 10-fold CV the improvements are also significant, except for targets on the test set. Indeed, standard deviations on the 10-fold CV test sets are always much higher compared to the dev set or to the test sets of 4-fold CV.