A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes
Table 3: Results on the atomic and full datasets.
['[BOLD] Dataset', '[ITALIC] atomic  [BOLD] Val. acc', '[ITALIC] atomic  [BOLD] BLEU', '[ITALIC] atomic  [BOLD] Moses', '[ITALIC] full  [BOLD] Val. acc', '[ITALIC] full  [BOLD] BLEU']
In particular, we observe bigger gains for java projects such as CoreNLP and guava. We hypothesize this is because program differences in Java tend to be longer than the rest. While this impacts on training time, at the same time it allows the model to work with a larger vocabulary space. On the other hand, our model performs similarly to MOSES for the node and slightly worse for the youtube-dl. A detailed inspection of the NL messages for node showed that many of them exhibit a fixed pattern in their structure. We believe this rigidity restrains the generation capabilities of the decoder, making it more prone to memorization. Despite the small data increase, we also trained our model on full datasets as a way to confirm the generative power of our model. In particular, we wanted to test the model is able leverage on atomic data to also capture and compress multi-file changes.