Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
Table 1: Dev accuracy results for MT-DNN model with different training set sampling strategies.
['[BOLD] Strategy', '[BOLD] % of Training Data 0.1%', '[BOLD] % of Training Data 1%', '[BOLD] % of Training Data 10%']
As an additional experiment, we used the learned difficulty parameters to compare data sampling strategies for a state-of-the-art NLI model, MT-DNN Liu et al. We sampled training data for SNLI at several intervals (0.1%, 1%, 10%) and trained the MT-DNN model with the sampled data. We trained each model, as well as the random sample baseline, using the publicly available MT-DNN code. Note that we report two random baselines: (i) those reported in the original work, which were obtained by training the MT-DNN model with a batch size of 32. Due to GPU resource constraints we had to train each MT-DNN model with a batch size of 8, and therefore report our reproduced random baseline results that we obtained as well (“Random (small batch)”). For very small samples of data, the AVI strategy outperforms random sampling and all other methods as well. As more data is sampled, the random models perform better. This indicates that a more advanced sampling strategy that starts with AVI then incorporates outliers (very easy/hard examples) at certain thresholds may improve learning as well.