Constructing a Highlight classifier with an Attention based LSTM Neural Network
Table 2: Experimental results of Gradient Boosting Highlight Classifiers (LightGBM). All values shown are for 7-fold ensemble models. Classes of models are separated by horizontal lines. Best scores form each section (overall) are in bold (underlined). Feature notation class 1: (cv/tfidf)(ab) - count vectorization/tfidf weighting with a/b: minimum/maximum n-grams. Class 2: (m/tfidf)(embedding) - mean/tfidf weighting of word embedding vectors as described in Eq. 12 with GloVe/BlazingText embeddings. Class 3: (cm/tfidf)(embedding) - count vectorization/tfidf weighting of stemmed unigrams.
['[EMPTY]', 'roc_auc', 'prec', 'rec']
Precision, recall and ROC AUC scores are illustrated for h-coverages of 0.2, 0.5, 0.8 and as continuous functions of document lengths. Test dataset described in Figure. Tables Both tables show detailed comparisons of the various models as well as the scale invariant and imbalance-robust metric: ROC AUC. When weighted against the number of features in the models - 300 for BlazingText, 50 for GloVe [glove1, glove2] and 50,000 - 1M for BOW models - the performance is impressive. We no longer have the model relying on individual words, but instead on word classes (clusters of similar context). The character of each of these clusters is projected onto one of the 50 (300) latent dimensions of our vector space, which more often than not, don’t correspond to a human interpretable semantic meaning. Because some words exist with multiple variations, depending on the grammar, it is sometimes good practice to normalize them to their absolute roots. For example, the words playing, players, played all become play. As far as the classifier is concerned its performance will be better when there is a single feature for N variations, rather than N features for each variation. Since our classifiers improve when stemming is applied, it indicates we are doing just this - reducing the number of useless features into fewer, stronger ones.