Adapting Neural Text Classification forImproved Software Categorization
TABLE I: Overview of the results. The column Train indicates the training set used: we always use both code-only and code-description function representations from the training set. Test is the test set: cd means code-description, co means code-only. Recall from Section V-A that cd simulates the situation when we have descriptions, and co simulates when we only have code, but both cd and co test sets have the same functions from the holdout set. The top four columns are results simulating when no description data is available (co). The algorithm lr 1.8k is the linear baseline using vocab size of 1.8k, nn+w is the neural architecture with Wikipedia embedding, nn+co is the code-only embedding, and nn+cd is the code-description embedding. P, R, and F correspond to precision, recall, and f-measure. run is the run number associated with the downloadable model and full output available in our online appendix; we include it to assist replication of our experiments and use of the models we created (see Section IX).
['Train', 'Test', 'Algo.', 'Standard P', 'Standard R', 'Standard F', 'Standard run', 'Challenge P', 'Challenge R', 'Challenge F', 'Challenge run', 'Libs P', 'Libs R', 'Libs F', 'Libs run']
These lines correspond to the linear regression for code-only (line 1) and code-description (line 5) data. In line 5, the F1 score on the standard dataset is 65, and precision/recall is 68/64. These results are broadly similar to the 67% F1 score reported by Wang et. when using text description data (mined from the web) with linear regression to classify Java projects from SourceForge (though the results are not directly comparable due to different categories, datasets, and programming language). However, when text data is not available, we observe a drop in performance to 34% F1 score, 43/33 precision/recall – a nearly 50% performance penalty as measured by F1 score. It achieves 78% F1 score with descriptions, compared to 46% without – over a 40% performance drop. Challenge results are even more extreme, from 40 to 17 F1 score. Interestingly, performance on the libs dataset is relatively stable, even with slightly improved results when description data is removed: from 48 to 52. One likely explanation is that the library functions could be called by functions in the training set, which would lead to the model learning to recognize those libraries as part of a category. Still, on the standard and challenge datasets, there is evidence for a large performance gap between code-only and code-description data.