Contextualized Code Representation Learning for Commit Message Generation
Table 1: Comparison of our proposed framework with baseline models using different evaluation metrics.
['[BOLD] Model  [BOLD] Baselines', '[BOLD] Model NMT', '[BOLD] BLEU-4 14.17', '[BOLD] ROUGE-1 21.29', '[BOLD] ROUGE-2 12.19', '[BOLD] ROUGE-L 20.85', '[BOLD] METEOR 12.99']
CoreGen outperforms baseline models across all evaluation metrics with at least 28.18%, 26.12% and 17.82% improvement on BLEU-4, ROUGE-L and METEOR scores, respectively. We attribute this to its ability to capture code contextual information for representing the most critical segments of code snippets, i.e., the changed code fragments. Besides, comparing to PtrGNCMsg that employs a pointer-generator network to tackle the OOV problem, CoreGen’s superior performance further supports our claim that exploiting contextualized code representation can achieve more accurate prediction of context-specific tokens on downstream code commit message generation task, leading to an elegant solution to OOV issue. Specifically, CoreGen II’s substantial improvement compared with the baselines also demonstrates Transformer’s strengths over traditional LSTM-based encoder-decoder models for the code commit message generation task. However, a remarkable performance gap still exists between CoreGen II and CoreGen, which, again, affirms the necessity of contextualized code representation learning.