Latent Predictor Networks for Code Generation
Table 3: BLEU and Accuracy scores for the proposed task on two in-domain datasets (HS and MTG) and an out-of-domain dataset (Django).
['[EMPTY]', '[BOLD] MTG bleu', '[BOLD] MTG xxAcc', '[BOLD] HS bleu', '[BOLD] HS xxAcc', '[BOLD] Django bleu', '[BOLD] Django xxAcc']
Regarding the retrieval results (cf. “Retrieval” row) , we observe the best BLEU scores among the baselines in the card datasets (cf. “MTG” and “HS” columns). A key advantage of this method is that retrieving existing entities guarantees that the output is well formed, with no syntactic errors such as producing a non-existent function call or generating incomplete code. As BLEU penalizes length mismatches, generating code that matches the length of the reference provides a large boost. The phrase-based translation model (cf. “Phrase” row) performs well in the Django (cf. “Django” column), where mapping from the input to the output is mostly monotonic, while the hierarchical model (cf. “Hierarchical” row) yields better performance on the card datasets as the concatenation of the input fields needs to be reordered extensively into the output sequence. Finally, the sequence-to-sequence model (cf. “Sequence” row) yields extremely low results, mainly due to the lack of capacity needed to memorize whole input and output sequences, while the attention based model (cf. “Attention” row) produces results on par with phrase-based systems. Finally, we observe that by including all the proposed components (cf. “Our System” row), we obtain significant improvements over all baselines in the three datasets and is the only one that obtains non-zero accuracies in the card datasets.