SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring
Table 2: Experimental results of all compared models on the ASAP dataset. Best result is in bold and 2nd best is underlined. Results are sorted by average performance. † denotes our implementation of a model from [Taghipour and Ng2016], ϕ denotes the baseline for statistical significance testing, ∗ denotes statistically significant improvement. ⋆ denotes non deep learning baselines.
['ID', 'Model', 'Dataset / Prompts 1', 'Dataset / Prompts 2', 'Dataset / Prompts 3', 'Dataset / Prompts 4', 'Dataset / Prompts 5', 'Dataset / Prompts 6', 'Dataset / Prompts 7', 'Dataset / Prompts 8', 'Average']
First, it is clear that the mean pooling is significantly more effective as compared to the last LSTM output. In the last setting, the performance of RNN is significantly worse compared to LSTM and GRU possibly due to the weaker memorization ability. However, the performance of LSTM, GRU and RNN are similar using the mean pooling setting. This is indeed reasonable because the adoption of a mean pooling layer reduces the dependency of the model’s memorization ability due to implicit access to all intermediate states. Overall, we observe that the performance of LSTM and GRU is quite similar with either mean pooling or last setting. Finally, we note that the performance of CNN is considerably better than RNN-based models. We also observe that a multi-layered LSTM performs considerably better than a single-layered LSTM. We also observe that adding layers also increases the performance. On the other hand, the bidirectional LSTM did not yield any significant improvements in performance. The performance of ATT-LSTM is notably much higher than the base LSTM. The best performing LSTM model is a multi-layered LSTM with 4 layers. Finally, we compare SkipFlow LSTM with deep learning models First, we observe that a higher dimensionality improves performance over d=50. Our SkipFlow LSTM (Tensor) outperforms LSTM†(d=300) significantly by 5%. The performance of LSTM†(d=300) and GRU†(d=300) are in fact identical and are only slightly better than feature engineering baselines such as EASE (BLRR). We also observe that ATT-LSTM and ML-LSTM (L=4) with both d=50 also consistently outperform LSTM† and GRU†. Conversely, our SkipFlow LSTM (Tensor) model outperforms the best feature engineering baseline (EASE) by about 6%.