The SOFC-Exp Corpus and Neural Approachesto Information Extraction in the Materials Science Domain
Table 6: Experiments: entity mention detection and typing. Results on test set (experiment-describing sentences only) in terms of F1, rightmost column shows the macro-average.
['[BOLD] Model', 'Exp.', 'Mat.', 'Val.', 'Dev.', '[BOLD] avg.']
Models are trained and results are reported on the subset of sentences marked as experiment-describing in the gold standard, amounting to 4,590 entity mentions in total. The CRF baseline achieves comparable or better results than the Bi-LSTM with word2vec and/or mat2vec embeddings. However, adding subword-based embeddings (bpe and/or BERT) significantly increases performance of the BiLSTM, indicating that there are many rare words. Again, the best results are obtained when using BERT or SciBERT embeddings or when using the original SciBERT model. It is relatively easy for all model variants to recognize Value as these mentions usually consist of a number and unit which the model can easily memorize. Recognizing the types Material and Device, in contrast, is harder and may profit from using gazetteer-based extensions.