IntelliCode Compose: Code Generation using Transformer
Table 5: Detailed evaluation results for various multilingual modeling approaches based on GPT-C. Model performance metrics are reported on multilingual test sample.
['[BOLD] Model', '[BOLD] PPL', '[BOLD] ROUGE-L Precision', '[BOLD] ROUGE-L Recall', '[BOLD] Edit similarity (%)', '[BOLD] Model size']
To prepare a multilingual version of the IntelliCode Compose, we have extracted a shared sub-token vocabulary for Python, C#, JavaScript and TypeScript programming languages using the BPE tokenizer. We explored and compared four following ways of training multilingual GPT-C models: Language-agnostic baseline completely disregards the language type information, effectively attempting to train the model as monolingual. We have found this approach to underperform significantly as compared to the monolingual versions for each language. Language-type embedding. Nlang denotes the number of programming languages in the training dataset. Language specific control codes, or prefixes, is an approach introduced in [T5, CTRL] that facilitates constrained language modeling. In what follows, for each programming language we insert a sequence of tokens in the beginning of each training sample according to: "lang ∗ remaining token sequence", where lang∈{Python,C#,JavaScript,TypeScript}. In expectation, control codes would signal the neural network that a given sequence belongs to a particular programming language. As shown in Tab. this approach works rather well, yielding results comparable with monolingual counterparts. Finally, we add a programming language classification task during model pretraining, an approach inspired by natural language inference (NLI) RocStories and SWAG tasks [mostafazadeh-etal-2016-corpus, zellers2018swag]. As such, the model is trained using two optimization objectives: language modeling and multiple choice classification to detect programming language.