CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Table 2: Results on natural language code retrieval. Baselines include four joint embeddings (first group) of NL and PL, RoBERTa, and RoBERTa which is continuously trained with masked language modeling on codes only (second group). PT stands for pre-training. We train CodeBERT (third group) with different settings, including using different initialization ( from scratch (Init=scratch) or initialized with the parameters of RoBERTa) and using different learning objectives (MLM, RTD, or the combination of both).
['model', 'ruby', 'javascript', 'go', 'python', 'java', 'php', 'Ma-Avg']
The first four rows are reported by Husain et al. NBoW represents neural bag-of-words. We train all these pre-trained models by regarding codes as a sequence of tokens. We also continuously train RoBERTa only on codes from CodeSearchNet with masked language modeling. Results show that CodeBERT consistently performs better than RoBERTa and the model pre-trained with code only. CodeBERT (MLM) learned from scratch performs better than RoBERTa. Unsurprisingly, initializing CodeBERT with RoBERTa improves the performance.