Constructing a Highlight classifier with an Attention based LSTM Neural Network
Table 3: Experimental results of LSTM Highlight Classifiers (TensorFlow). All values shown are for 7-fold ensemble models. Classes of models are separated by horizontal lines. Best scores form each section (overall) are in bold (underlined). Baseline model is a 128 hidden unit, single layer LSTM, with 64 batch size. g/f designates usage of GloVe/BlazingText embeddings, a - attention mechanism, b - bi-directionality, p - use of punctuated dataset for training.
['[EMPTY]', 'roc_auc', 'prec', 'rec']
Precision, recall and ROC AUC scores are illustrated for h-coverages of 0.2, 0.5, 0.8 and as continuous functions of document lengths. Test dataset described in Figure. Tables Both tables show detailed comparisons of the various models as well as the scale invariant and imbalance-robust metric: ROC AUC. The improvement in performance exists regardless of the LSTM variant used. Among the LSTM variants themselves, we point out some obvious trends. Bidirectional LSTMs are very useful where the context of the input is needed, for example in situations where performance might be enhanced by knowledge of what comes before and after a specific point in time. We see no increase from bi-directionality at all, and in fact, in some cases we see a small decline in performance. This could occur due to the increased complexity of the network and the difficulty in reaching the local minimum. If we also account for practical aspects, the training time doubles and saturation occurs much slower. Training a distributed representation of words has been shown before to be very sensitive to the corpora it is trained on. [embeddings2, embeddings3] Since every vector encodes semantic relationships among words, it is the context of those words which decides their representation. Training an embedding space on corpora about biology might show a high similarity score between ”apple” and ”pear” while this might not be the case if trained on a more technological corpus. In a similar manner, the custom-trained embedding space will better capture the unique terminology to the market research corpora. The performance increase is most significant in terms of precision, increasing from 0.72 to 0.78, while recall increases from 0.70 to 0.73.