The SOFC-Exp Corpus and Neural Approachesto Information Extraction in the Materials Science Domain
Table 5: Experiments: identifying experiment-describing sentences. P, R and F1 for experiment-describing sentences. With the exception of SVM, we downsample the non-experiment-describing sentences of the training set by 0.3.
['Model', '[BOLD] dev F1', '[BOLD] test P', '[BOLD] test R', '[BOLD] test F1']
The neural models with byte-pair encoding embeddings or BERT clearly outperform the SVM and logistic regression models. Within the neural models, BERT and SciBERT add the most value, both when using their embeddings as another input to the BiLSTM and when fine-tuning the original BERT models. Note that even the general-domain BERT is strong enough to cope with non-standard domains. Nevertheless, models based on SciBERT outperform BERT-based models, indicating that in-domain information is indeed beneficial. For performance reasons, we use BERT-base in our experiments, but for the sake of completeness, we also run BERT-large for the task of detecting experiment sentences. Because it did not outperform BERT-base in our cross-validation based development setting, we did not further experiment with BERT-large. However, we found that it resulted in the best F1-score achieved on our test set. In general, SciBERT-based models provide very good performance and seem most robust across dev and test sets. Overall, achieving F1-scores around 67.0-68.6, such a retrieval model may already be useful in production. However, there certainly is room for improvement.