Ask, and shall you receive?: Understanding Desire Fulfillment in Natural Language Text
Table 4: Test set performances. Our structured model, LSNM, outperforms the unstructured, TE and BoW models.
['[BOLD] Data', '[BOLD] Model type', '[BOLD] Name', '[BOLD] P', '[BOLD] R', '[BOLD] F']
For evaluation, we compared test set performances using F1 score of the positive (desire fulfilled) class. We also included a simple Logistic Regression baseline based on Bag-of-Words (BoW) features. For training the unstructured model, we experimented with different algorithms and show the results for the best two models: LR (Logistic Regression) and DT (Decision Trees). We report median performance values over 100 random restarts of our model since its performance depends on the initialization of the weights. Also, our model requires the number of latent states, H, as input which was set to be 2 and 15 for the MCTest and SimpleWiki datasets respectively using cross-validation. The difference in optimal H values (and F1 scores) for the two datasets could be attributed to the difference in complexity of the language and concepts used in them. The MCTest dataset consists of children stories, focusing on simple concepts and goals (e.g., ‘wanting to go skating’) and their fulfillment is indicated explicitly, in simple and focused language (e.g., ‘They went to the skating rink together.’). On the other hand, SimpleWiki describes real-life desires (e.g., ‘wanting to conquer a country’), which require sophisticated planning over multiple steps, which may provide only indirect indication of the desire fulfillment status. This added complexity resulted in a harder classification problem, and increased the complexity of inference over several latent states.