CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Table 5: Results on Code-to-Documentation generation, evaluated on CodeSearchNet with smoothed BLEU-4 score.
['model', 'ruby', 'javascript', 'go', 'python', 'java', 'php', 'overall']
As we can see, models pre-trained on programming language outperform RoBERTa, which illustrates that pre-trainning models on programming language could improve code-to-NL generation. Besides, results in the Table show that CodeBERT pre-trained with RTD and MLM objectives brings a gain of 1.3% BLEU score over RoBERTa overall and achieve the state-of-the-art performance on the majority of programming languages. These results show that our pre-trainning objectives (MLM and RTD) are effective for CodeBERT on code-to-NL generation tasks.