When to reply? Context Sensitive Models to Predict Instructor Interventions in MOOC Forums
Table 1: Prediction performance for the positive class (intervened threads) of the three proposed models versus a neural baseline hLSTM, using hierarchical encoding of the discussion thread. Best performance in F1 and Recall is bolded. Column 2 shows # of threads in each course and the intervention ratio, the ratio of intervened to non-intervened threads, in parentheses.
['[BOLD] Course', '[BOLD] Thread  [BOLD] Demographics', '[BOLD] hLSTM (Baseline) P', '[BOLD] hLSTM (Baseline) R', '[BOLD] hLSTM (Baseline)  [ITALIC] F1', '[BOLD] UPA P', '[BOLD] UPA R', '[BOLD] UPA  [ITALIC] F1', '[BOLD] PPA P', '[BOLD] PPA R', '[BOLD] PPA  [ITALIC] F1', '[BOLD] APA P', '[BOLD] APA R', '[BOLD] APA  [ITALIC] F1']
Model hyperpameter settings. All proposed and baseline neural models are trained using Adam optimizer with a learning rate of 0.001. We used cross-entropy as loss function. Importantly we updated the model parameters during training after each instance as in vanilla stochastic gradient descent; this setting was practical since data on most courses had only a few hundred instances enabling convergence within a reasonable training time of a few hours (see Models were trained for a single epoch as most of our courses with a few hundred thread converged after a single epoch. We used 300-dimensional GloVe vectors and permitted the embeddings to be updated during the model’s end-to-end training. The hidden dimension size of both lstmpost and lstmctx are set to 128 for all the models. Our models of UPA, PPA individually better the baseline by 5 and 2% on F1 and 3 and 6% on recall respectively. UPA performs the best in terms of F1 on average while PPA performs the best in terms of recall on average. At the individual course level, however, the results are mixed. UPA performs the best on F1 on 5 out of 12 courses, PPA on 3 out 12 courses, APA 1 out of 12 courses and the baseline hLSTM on 1. PPA performs the best on recall on 7 out of the 12 courses. We also note that course level performance differences correlate with the course size and intervention ratio (hereafter, i.ratio), which is the ratio of intervened to non-intervened threads. UPA performs better than PPA and APA on low intervention courses (i.ratio ⪅ 0.25) mainly because PPA and APA’s performance drops steeply when i.ratio drops (see col.2 parenthesis and F1 of PPA and APA). While all the proposed models beat the baseline on every course except casebased-2. On medicalneuro-2 and compilers-4 which have the lowest i.ratio among the 12 courses none of the neural models better the reported baseline Chandrasekaran et al. (course level not scores not shown in this paper). The effect is pronounced in compilers-4 course where none of the neural models were able to predict any intervened threads. This is due to the inherent weakness of standard neural models, which are unable to learn features well enough when faced with sparse data.