Exploring Software Naturalness throughNeural Language Models
Table 4: Test set accuracy for the VI task on the full and reduced ("red") datasets; C-BERT model id indicates tokenization (C|K|S), LM masking objective (M|W) and aggregation(T|B); Aggr indicates the aggregation method
['[BOLD] Model', '[BOLD] LM + Tokenizer', '[BOLD] Masking', '[BOLD] Aggr.', '[BOLD] FFmpeg  [ITALIC]  [BOLD] full', '[BOLD] FFmpeg  [ITALIC]  [BOLD] red', '[BOLD] QEMU  [ITALIC]  [BOLD] full', '[BOLD] QEMU  [ITALIC]  [BOLD] red', '[BOLD] Combined  [ITALIC]  [BOLD] full', '[BOLD] Combined  [ITALIC]  [BOLD] red']
C-BERT models, with aggregation outperform the strongest GGNN baseline by a reasonable margin of 3-4 points across all datasets. They also perform better than BiLSTM and CNN baselines on both the full and reduced datasets.