StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow
Table 3. Comparison of BiV-HNN and its variants.
['Model', 'Python Testing Set Prec.', 'Python Testing Set Rec.', 'Python Testing Set  [ITALIC] F1', 'SQL Testing Set Prec.', 'SQL Testing Set Rec.', 'SQL Testing Set  [ITALIC] F1']
Due to space constraints, we do not show the accuracy of each model, which has roughly the same pattern as F1. We have made the following observations: (1) Single-view variants. BiV-HNN outperforms Text-HNN and Code-HNN by a large margin on both datasets, showing that both views are critical for our task. In particular, by incorporating code content information, BiV-HNN is able to improve Text-HNN by 7% on Python dataset and around 5% on SQL dataset in F1. (2) No-query variant. On Python dataset, the integration of the question information in BiV-HNN brings 3% F1 improvements over BiV-HNN-nq, which shows the effectiveness of associating the question with the code snippet for identifying code answers. For SQL dataset, adding the question gives no obvious benefit, possibly because the code content in each SQL program already carries critical information for making a prediction (e.g., a SQL program containing the command keyword “SELECT” is very likely to be a solution to the given question, regardless of the question content). (3) “Flat”-structure variants. On both datasets, the hierarchical structure leads to 1%∼2% improvements against the “flat” structure in both bi-view (BiV-HNN vs. BiV-RNN) and single-view setting (Text-HNN vs. Text-RNN). (4) Non-sequence variant. On Python dataset, BiV-HNN outperforms BiV-HFF by around 2%, showing the block-level Bi-GRU is preferable over feedforward neural networks. The two models get roughly the same performance on SQL, probably because our task is easier in SQL domain than in Python domain as we mentioned earlier.