Adabot: Fault-Tolerant Java Decompiler
Table 3: Performance of attention-based NMT and Transformer models on the purified and redundant dataset. We evaluate each task with both BLEU-4 and WER.
['TASK', 'BLEU-4(%)', 'WER(%)']
However, Transformer completely outweighs attention-based NMT on the redundant dataset. It is because even if attention is used as an auxiliary measure to alleviate the vanishing gradient problem which is severe when dealing with long sequences, it is only powerful enough to help it learn the structural information but not the lexical information. Therefore, we can concluded that attention mechanism is lot better in handling the unbalanced distribution of information in programming languages compared with recurrence mechanism, which makes Transformer a more suitable model for Java decompilation task than attention-based NMT. And WER is more sensitive to its poor learning on the lexical information since WER based its evaluation on substitution, deletion and insertion of words in a sequence. Conclusively, WER is a more sensitive and informative metric for the reverse engineering task than BLEU-4.