Predicting Human Activities from User-Generated Content
Table 11: Per-class accuracy (%) @ keval and ACR scores for the 806-class prediction task. Note that removing h from either fullT or fullA gives the same model. For ACR only, lower is better.
['[ITALIC] keval', '1', '2', '3', '5', '10', '25', '50', '75', '100', '200', '300', '[BOLD] ACR']
This version of the task also simulates a more realistic scenario, since predictions can be made for the “other” class when the model does to expect the user to claim to do an activity from any of the known clusters. In this setting, we see that the fullT−p model works well for keval≤25, suggesting that the use of the attribute vectors helps, especially when predicting the correct cluster within the top 25 is important. For keval≥50, the same fullT−a,p model that worked best in the 50-class setup again outperforms the others. Here, in contrast to the 50-class setting, using the full set of tweets usually performs better than focusing only on the human activity content. Interestingly, the best ACR scores are even lower in the 806-class setup, showing that it is just as easy to rank users by their likelihood of writing about an activity, even when considering many more activity clusters.