Learning Eligibility in Cancer Clinical Trials using Deep Neural Networks
Table 3: Overall results on the validation set for all the classifiers using a dataset of 106 samples and the full dataset (4.1 × 106) samples. Both experiments were performed using 20% of the prototypes for validation and 80% for training. The best results are marked in bold.
['[BOLD] Classifier', '[BOLD] Dataset Size', '[BOLD] Precision', '[BOLD] Recall', '[ITALIC] F1', '[BOLD] Cohen’s  [ITALIC] κ']
The F1 achieved when using 106 samples (800K for training) was 0.87 The Cohen’s Kappa coefficient of agreement between the predicted and the true labels in the validation set was κ=0.75, which is regarded as a substantial agreement. The results did not improve significantly when using the full dataset of 4.1 × 106 samples. In this case, the F1 achieved was 0.88 with a Cohen’s Kappa coefficient κ=0.76. The validation score converges with the training score, and the estimator does not benefit much from more training data, denoting a bias error. It has been reported that the phenomenon of not being able to increase the performance with additional data can be overcome with the use of deep learning models applied to complex problems, in contrast to a fast but thin architecture such as FastText (as will be proved later when using CNNs). On the contrary, the model did not suffer from a variance error. The model of the whole dataset, using 3,257,179 training examples, bigrams, and pre-trained word-embeddings, eventually yielded an accuracy of 0.91 for the validation set comprising 814,295 samples. The curve converged with the score obtained in the training sample to a maximum of 0.79 The validation score converges with the training score, and the estimator does not benefit much from more training data, denoting a bias error. The F1 achieved when using 106 samples (800K for training) was 0.79 The Cohen’s Kappa coefficient of agreement between the predicted and the true labels in the validation set was κ=0.57. The results did not improve when using the full dataset of 4.1 × 106 samples. In this case, the F1 achieved was 0.79 with a Cohen’s Kappa coefficient κ=0.58. This estimator benefited the most, compared with the other models, from more training data. Nonetheless, as expected, the computational cost on prediction time was expensive, and using 106 samples was equivalent to 16 core-hours of CPU. The Cohen’s Kappa coefficient of agreement between the predicted and the true labels in the validation set was κ=0.83, which is regarded as an almost perfect agreement. The results did not improve significantly when using the full dataset of 4.1 × 106 samples. In this case, the F1 achieved was 0.93 with a Cohen’s Kappa coefficient κ=0.84.