Feature Assisted bi-directional LSTM Model for Protein-Protein Interaction Identification from Biomedical Texts
Table 5: Comparison of different activation functions with same hyperparameters values
['[BOLD] Activation Function', '[BOLD] F-Score (AiMed)', '[BOLD] F-Score (BioInfer)']
We setup all the experiments by varying the hyper-parameter values and analyze the behaviors of our model. For AiMed dataset, we observed that addition of LSTM units improves the model performance to a certain extent. Thereafter, it keeps on decreasing gradually. We define an optimal value 64 for the same, via cross-validation experiment. It was observed that deep MLP layer helps to improve the overall performance of the model when compared to a shallow MLP layer as shown in Figure 3. However, this improvement was dependent upon the size of the output layer, which increases initially from 20 to 30 and then starts decreasing while the size of the output layer is increased to 50. We also notice that stacking of another MLP layer makes our model complex, thus reducing the performance on a cross-validation setting. In case of BioInfer dataset, we also observe quite a similar trend in performance with the addition of LSTM units, size of MLP output layer and stacking of another MLP layer. We also analyze the performance of our model on the number of epochs for which training was performed on both the datasets. On AIMed dataset, the value of F1-score initially shows a short fall with the increasing number of epochs from 1 to 2 and then shows regular growth with the increasing number of epochs from 2 to 130, and finally a dip on further increasing the number of epochs to 135 and 140. This can be attributed to the fact that training on a very large number of epochs makes our model over-fitted and hence the cross-validation accuracy decreases. There has also been an initial decline in the F1-score in case of BioInfer dataset but then there has been steady increase with the increase in the number of epochs. We achieve the optimum performance with the same number of epochs (130) for both the datasets. To further compare the performance of ReLU with Sigmoid, we have also conducted the experiments considering both ReLU and Sigmoid on both the datasets. On both the datasets, the sigmoid function was found to be superior over ReLU. \captionof figureEffect of varying MLP layers and their output sizes \captionof figureEffect of varying epochs on F1-Score