Summarizing Dialogic Arguments from Social Media
Table 4: Results for classification on test set for each topic. Best performing model in bold.
['[BOLD] ID', 'Classifier', '[BOLD] Features', '[BOLD] Gun Control  [BOLD] F-weight Avg.', '[BOLD] Gun Control  [BOLD] F-weight Avg. Coref', '[BOLD] Gay Marriage  [BOLD] F-weight Avg.', '[BOLD] Gay Marriage  [BOLD] F-weight Avg. Coref', '[BOLD] Abortion  [BOLD] F-weight Avg.', '[BOLD] Abortion  [BOLD] F-weight Avg. Coref']
CNN + BiLSTM. Using this as a motivation, we include a convolutional layer and max pooling layer before the input is fed into an RNN. The model used for binary classification consists of a 1D convolution layer of size 3 and 32 different filters. The convolution layer takes as input the GloVe embeddings. The outputs of the LSTM are fed through a sigmoid layer for binary classification. LSTM creates a validation set by a 4 to 1 random selection on the training set. Regularization is performed by using a drop-out rate of 0.2 in the drop-out layer. Simple Ablation Models. The LexRank summarizer performs best across all topics, but overall the results show that summarizers aimed at newswire or monologic data do not work on argumentative dialog. Feature Combination Models. For the gun control topic, LIWC categories on the current sentence give an F-score of 0.72. Adding LIWC from the previous sentence improves it to 0.73 (rows 5 and 9, without coref column). In contrast, just doing a coref replacement improves LIWC current sentence score to 0.74 (row 5 for gun control, with and without coref columns). A paired t-test on the result vectors shows that coref replacement provides a statistically significant improvement at (p <0.04). For the Abortion topic, the overall performance is low as compared to the other two topics suggesting that arguments used for abortion are harder to identify. Both DAC, Word2vec scores are quite low but readability and LIWC do better. The LSTM models on their own do not perform better than SVM across topics, but adding features to the LSTM models improves them beyond the SVM results. Paired t-tests on the result vectors show that the differences in F-score are statistically significant when we compare LSTM to LSTM with features for each topic (p <0.01) for all topics, indicating that adding contextual features makes a significant improvement. Adding LIWC categories from current and previous utterances to LSTM also improves performance for gun control and abortion. For the gay marriage topic, LSTM combined with LIWC and readability works better than LSTM alone.