Continuous Experience-aware Language Model
Table 4: Performance on identifying experienced users.
['[BOLD] Models', '[BOLD] NDCG', '[BOLD] Kendall Tau']
Identifying Experienced Users. Our second task is to find experienced members of this community, who have the potential of being citizen journalists. In order to evaluate the quality of the ranked list of experienced users generated by our model, we consider the following proxy measure for user experience. In NewsTrust, users have Member Levels determined by the NewsTrust staff based on community engagement, time in the community, other users’ feedback on reviews, profile transparency, and manual validation. We use these member levels to categorize users as experienced or inexperienced. Here we consider the top-performing baseline models from the previous task. We report the Normalized Discounted Cumulative Gain (NDCG) and the Normalized Kendall Tau Distance for the ranked lists of users generated by all the models. The better model should exhibit higher NDCG, and lower Kendall Tau Distance.