Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
Table 3: Examples from the SNLI and SSTB data sets where the ranking in terms of difficulty varies widely between human and DNN models. In all cases difficulty is ranked from easy to hard (1=easiest).
['Task', 'Label', 'Item Text', 'Difficulty ranking Humans', 'Difficulty ranking LSTM', 'Difficulty ranking NSE']
S6SS0SSS0Px2 Analysis of Differences That is, we calculated the absolute difference in ranking between the human model and the DNN model, and selected those where that value was highest. The average absolute difference in ranking was around 40 for the SNLI task and around 30 for SSTB, for both the LSTM and NSE ensembles. We can see interesting patterns in the discrepancies. Upon inspection of the gathered labels, the high difficulty comes from the fact that there were many Turkers who labeled the data as neutral and also many who labeled it as contradiction. For SSTB, we see similar patterns For humans, one of the easiest review snippets is clearly positive (row 3), mainly because we know who Anthony Hopkins is and know how to rate his quality as an actor. However for the DNN models, the text itself does not have a lot of positive or negative signal and therefore the item is considered very difficult. On the other hand, the last example is very difficult for humans (row 4), possibly due to the relatively neutral text. However, for the DNN models certain terms such as “stultifyingly contrived” may signal a more negative review and lead to the item being easier.