Scientific Statement Classification over arXiv.org
Table 1: Labeled data for “13 nest” classification task
['[BOLD] Class', '[BOLD] Included Members', '[BOLD] Frequency']
First, some classes were strongly separable in the task posed as-is, such as acknowledgement, abstract and proof, at near-perfect classification rates. Second, there were “confusion nests” of interconnected classes. Most notably, proposition, lemma and theorem, dominated by the latter two, had a strong indication of a shared language nest. It stands to reason that as a first approximation we can then unify this constellation of classes into a single parent class, which we named after the most abstract label in the group - proposition. Such regroupings simplify the task and reduce the classification difficulty when performed correctly. Lastly, we also remark that the model performs in a very scattershot manner on about half of the label set. In some cases that is due to very little training data (e.g. hint), in others it is due to limitations of the task setup (e.g. experiment, which is hard to separate from example and result without additional context). Following these observations, we propose a reduced task with an emphasis on class-separability at scale. To this end, we preserve the clearly separable cases and group the observed inter-confused nests together into more abstract union classes. All low-volume and scattershot classes are ignored for the reduced task. This brings us to a “13 nest” classification task, based on 25 of the original 50 classes, grouped into 13 separable classes. Importantly, we retain 99% of the available data, or 10.4 million from the original 10.5 million paragraphs.