Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
Table 1: Probing of the LSTM model at intermediate timesteps for the Digit Sum dataset. Left column is the input and underlined digit emphasizes the last input digit. Ground Truth is the running sum up to that point. Predicted values by the LSTM models are in prediction column. The number in parantheses are the standard deviation. The intermediate representation of Baby Step curriculum model is closer to running sum of the input sequence.
['Input Sequence', 'Ground Truth', 'Baby Steps Curriculum Prediction', 'One-Pass Curriculum Prediction', 'Sorted Prediction', 'No-CL Prediction']
Probing Internal Model Representations. We analyze the behavior of the model by using the intermediate representations during processing of a sequence. As we discussed previously, we feed the hidden representations of each digit to the regression node to predict at each timestep of the input. The prediction of the model trained with One-Pass curriculum and Sorted Baseline shows no correlation with the running sum of the digits. The Baby Step curriculum model is able to predict similar values to running sum.