Recurrent Neural Network Encoder with Attention for Community Question Answering
Table 2: The RNN encoder results for cQA tasks (bold is best).
['Model', 'Task A MAP', 'Task A F1', 'Task B MAP', 'Task B F1', 'Task C MAP', 'Task C F1']
We observe that the attention model always gets better results than the RNN without attention, especially for task C. However, the RNN model achieves a very low F1 score. For task B, it is even worse than the random baseline. We believe the reason is because for task B, there are only 2,670 pairs for training which is very limited training for a reasonable neural network. For task C, we believe the problem is highly imbalanced data. Since the related comments did not directly comment on the original question, more than 90% of the comments are labeled as irrelevant to the original question. The low F1 (with high precision and low recall) means our system tends to label most comments as irrelevant. In the following section, we investigate methods to address these issues.