Recommendations for Datasets for Source Code Summarization
Table 1: Average BLEU Scores from 15 epochs for each of the four sets.
['[BOLD] SplittingStrategy', 'Set1', 'Set2', 'Set3', 'Set4']
We observe a large “false” boost in BLEU score when split by function instead of split by project We consider this boost false because it involves placing functions from projects in the test set into the training set – an unrealistic scenario. An average of four runs when split by project was 17.41 BLEU, a result relatively consistent across the splits (maximum was 18.28 BLEU, minimum 16.10). Our conclusion is that splitting by function is to be avoided during dataset creation for source code summarization. Beyond this narrow answer to the RQ, in general, any leakage of information from test set projects into the training or validation sets ought to be strongly avoided, even if the unit of granularity is smaller than a whole project.