Is artificial data useful for biomedical Natural Language Processing algorithms?
Table 5: Phenotyping results for CNN and Naive Bayes (NB), test-pheno. Best performing models for CNN data augmentation experiments are highlighted in bold. We report results for the models trained with: real data augmented with generated gen data, real data only, 2 × real data upsampled twice, gen data only, gen-key data without traces of the input real data.
['[EMPTY]', 'Obesity', 'Non Adherence', 'Developmental.Delay.Retardation', 'Adv. Heart Disease', 'Adv. Lung Disease', 'Schizo and other Psych. Disorders', 'Alcohol Abuse', 'Other Substance Abuse', 'Chr. Pain Fibromyalgia', 'Chr. Neurological Dystrophies', 'Adv. Cancer', 'Depression', 'Dementia', '[BOLD] avg.']
Phenotype Classification. They indicate that the artificial training data used as complimentary to the real training data is in general beneficial for the CNN model (e.g., av. F-score=0.50 for real + gen >0.47 for real). real +gen setup also outperforms the model trained using larger volume data, where the training data was repeated two times (2 × real). Overall, real +gen outperforms real for 9 phenotypes out of 13 with an average ΔF-score=0.06, while 2 × real for 6 phenotypes with an average ΔF-score=0.04 only. To get further insights into the actual informativeness of the generated data, we study the performance of both CNN and NB in a series of setups where the artificial training data fully replace the real training data. To be more precise, we study: ( a) gen setup, where the full generated data with traces of input key phrases are used as the training data; and (b) gen-key setup, where the generated text without traces of input data is used as the training data (see They show that average performances of gen and real tend to be comparable for each algorithm (e.g., Δ avg. F-score=0.03 for both CNN and NB). The gen-key setup results in a significant performance drop (of F-score=0.2 on average). However, the gen-key text still potentially bears some relevant information that allows both CNN and NB have comparable performance for this setup.