Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection
Table 2: The evaluation result for key term extraction.
['[BOLD] Model (a) Oracle', '[BOLD] Model (a) Oracle', '[BOLD] MAP (%) 77.2', '[BOLD] P@R (%) 77.2']
Row (a) is the oracle score, which is for reference. Since we only selected 1,000 most frequent key terms as candidates from the training set, we can’t achieve 100% accurate performance. The score of baseline approaches we applied are in rows (b) to (d), and rows (e), (f) are the performance of the proposed neural attention model. The supervised learning baselines outperformed the Tf-idf Sorting baseline (rows (c), (d) vs (b)). That is because without supervised learning, we may not fit the dataset, and we also can’t predict the key terms which do not exist in the document. Besides, like the experiment we previously conducted, LSTM shows better ability of handling sequence information in comparison to original neural networks (rows (d) vs (c)), so the LSTM network performs better while using both MAP and P@R as evaluation methods. We found that the performance of our neural attention model with sharpening-attend mechanism degraded while comparing to the original LSTM (rows (e) vs (d), but the one with smoothing attention outperformed all the other approaches (rows (f) vs (b), (c), (d), (e)). This result proved that adding more relevant elements into consideration can help solving sequence classification problems.