Constructing a Highlight classifier with an Attention based LSTM Neural Network
Table 4: Experimental results of the four sampling algorithms evaluated with the two best performing GBM and LSTM models from Tables 2 and 3. Precision/Recall scores are illustrated for h-coverages of 0.2, 0.5, 0.8 and document lengths < 200 words, 200-500 words, and above 500 words. Test dataset described in Figure. 5.
['[EMPTY]', '[innerwidth=2cm]h-covdoc len', 'LSTM Models <200 words', 'LSTM Models 200-500 words', 'LSTM Models >500 words', 'GBM Models <200 words', 'GBM Models 200-500 words', 'GBM Models >500 words']
Another trend is that precision increases as documents get longer. This indicates that classifiers have good grasp of the concept of a highlight. Imagine hitting a target as the target gets larger in size. If you know where the center of the region with the highest score is located, as the target gets larger you will only hit this region more. It’s diagnostic ability is barely above that of chance level, as indicated in the ROC AUC plots (bottom). We can surmise that the seq11 sampling method has the lowest predictive power. The most reliable model with the highest precision, for all ranges of h-coverage and document length choose is then the hscore sampling method.