LSTM-based Deep Learning Models for non-factoid answer selection
Table 3: The experimental results of InsuranceQA for QA-LSTM, QA-LSTM/CNN and QA-LSTM with attentions
['[EMPTY]', '[BOLD] Model', '[BOLD] Validation', '[BOLD] Test1', '[BOLD] Test2']
In this section, detailed analysis on experimental results are given. From Row (A) to (C), we list QA-LSTM without either CNN structure or attention model. They vary on how to utilize the biLSTM output vectors to form sentential embeddings for questions and answers in shown in section 3.1. We can observe that just concatenating of the last vectors from both direction (A) performs the worst. It is surprised to see using max-pooling (C) is much better than average pooling (B). The potential reason is that the max-pooling extracts more local values for each dimension, so that more local information can be reflected on the output embeddings. We also investigate the proposed models on how they perform with respect to long answers. We divide the questions of Test1 and Test2 sets into eleven buckets, according to the average length of their ground truths. We select models of (C), (F), (H) Model (C) is without attention and sentential embeddings are formed only by max pooling. Model (F) utilizes CNN, while model (H) and (I) integrate attention. However, as the ground lengths increase, the gap between (C) and other models becomes more obvious. The similar phenomenon is also observed in the right figure for Test2. This suggests the effectiveness of the two extensions from the basic model of QA-LSTM, especially for long-answer questions.