Recommendations for Datasets for Source Code Summarization
Table 2: Number of method-comment pairs in the train, validation, test sets used in each random split set when split by project (BP) and by function (BF).
['[EMPTY]', 'BP Set1', 'BP Set2', 'BP Set3', 'BP Set4', 'BF All Sets']
Our methodology for answering RQ1 is to compare the results of a standard NMT algorithm with the dataset split by project, to the results of the same algorithm on the same dataset, except with the dataset split by function. We then use an off-the-shelf, standard NMT technique called attendgru provided pre-release by LeClair et al. The technique is just an attentional encoder/decoder based on single-layer GRUs, and represents a strong NMT baseline used by many papers. We train attendgru with each of the four training sets, find the best-performing model using the validation set associated with that training set (out of 10 maximum epochs), and then obtain test performance for that model. We report the average of the results over the four random splits. Note that we used the same configuration for attendgru as LeClair et al. report, except that we reduced the output vocabulary to 10k to reduce model size.