Natural- to formal-language generation using Tensor Product Representations
Table 2: Results of AlgoLisp dataset
['[BOLD] MODEL (%)', '[BOLD] Full Testing Set  [BOLD] Acc', '[BOLD] Full Testing Set  [BOLD] 50p-Acc', '[BOLD] Full Testing Set  [BOLD] M-Acc', '[BOLD] Cleaned Testing Set  [BOLD] Acc', '[BOLD] Cleaned Testing Set  [BOLD] 50p-Acc', '[BOLD] Cleaned Testing Set  [BOLD] M-Acc']
Generating Lisp programs requires sensitivity to structural information because Lisp code can be regarded as tree-structured. Given a natural-language query, we need to generate code containing function calls with parameters. Each function call is a relational tuple, which has a function as the relation and parameters as arguments. We evaluate our model on the AlgoLisp dataset for this task and achieve state-of-the-art performance. Each sample contains a problem description, a corresponding Lisp program tree, and 10 input-output testing pairs. We parse the program tree into a straight-line sequence of tuples (same style as in MathQA). AlgoLisp provides an execution script to run the generated program and has three evaluation metrics: the accuracy of passing all test cases (Acc), the accuracy of passing 50% of test cases (50p-Acc), and the accuracy of generating an exactly matching program (M-Acc). AlgoLisp has about 10% noisy data (details in the Appendix), so we report results both on the full test set and the cleaned test set (in which all noisy testing samples are removed). Ablation experiments with TP2LSTM and LSTM2TP show that, for this task, the TP-N2F Decoder is more helpful than TP-N2F Encoder. This may be because lisp codes rely more heavily on structure representations.