Be Consistent! Improving Procedural Text Comprehension using Label Consistency
Table 3: Comparing LaCE vs. ProStruct with varying amount of labeled paragraphs available per training topic. We compare their performance in terms of F1 on ProPara test partition.
['Models', 'Proportion of labeled paragraphs used per training topic', 'Proportion of labeled paragraphs used per training topic', 'Proportion of labeled paragraphs used per training topic']
We then trained the ProStruct and LaCE models varying two different parameters: (1) the percentage of the labeled (ProPara) training data used to train the system (2) for LaCE only, whether the additional unlabeled data was also used. This allows us to see performance under different conditions of sparsity of labeled data, and (for LaCE) also assess how much unlabeled data can help under those conditions. During training, the unused labeled data was ignored (not used as unlabeled data). We keep the dev and test partitions the same as original dataset, picking a model based on dev performance and report results on test partition. In the first two rows, ProStruct and LaCE are both trained with x% of labeled data, while the last row reports performance of LaCE when it also has access to new unlabeled paragraphs. In addition, unlabeled data adds an additional boost to this performance, in particular when labeled data is sparse. We believe that having access to unlabeled paragraphs and optimizing consistency across paragraphs for training topics, helps LaCE generalize better to unseen topics.