Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts
Table 5: Ablation analysis. F1 scores are reported. “− context” is our model without the context enriching layer. “− seq. opt.” is our model without the label sequence optimization layer. “− dropout reg.’ is our model using the standard dropout strategy without the expectation-linearization regularization. “− attention” refers to the model without attention-based pooling, i.e., in the sentence encoding layer, the final hidden state is used for the HSLN-RNN model while max-pooling is used for the HSLN-CNN model.
['2.5 Model', 'HSLN-RNN', 'HSLN-CNN']
When the context enriching layer is removed, both models experience the most significant performance drop and can only be on par with the previous state-of-the-art results, strongly demonstrating that this proposed component is the key to the performance improvement of our model. Furthermore, even without the label sequence optimization layer, our model still significantly outperforms the best published methods that are empowered by this layer, indicating that the context enriching layer we propose can help optimize the label sequence by considering the context information from the surrounding sentences. Last but not the least, the dropout regularization and attention-based pooling components we add to our system can help further improve the model in a limited extent.