Learning Invariant Representations of Social Media Users
Table 1: Reddit author ranking results with 111,396 possible targets. The best results for each feature group are in printed in bold. The proposed Invariant User Representations are denoted IUR, with variations of the full model noted in parenthesis. MRR stands for the mean reciprocal rank, MR for median rank, and R@k stands for recall at the top k ranked episodes. Larger numbers are better (⇑) except for MR where lower rank is better (⇓). Metrics are computed over 25,000 queries.
['[BOLD] Input Features', '[BOLD] Method', '[BOLD] MRR (⇑)', '[BOLD] MR (⇓)', '[BOLD] R@1 (⇑)', '[BOLD] R@2', '[BOLD] R@4', '[BOLD] R@8']
Results. For each combination of features considered, the rankings based on the proposed IUR embeddings consistently outperform all methods considered, both neural and classical. We also report results on several variations of our model, noted in parenthesis. First, using the proposed architecture for fθ but the softmax loss results in ranking performance comparable to the baseline system. Second, using a recurrent architecture rather than self-attention to aggregate information across an episode results in significantly worse performance. Finally, omitting time features results in worse performance. Performance on novel users. To better understand the ability of the proposed embedding to generalize to new users, we performed a further evaluation in which authors were restricted to those not seen at training time. For the IUR incorporating all features, this yielded a MRR of 0.50, while our extension of \newciteshrestha2017convolutional obtains 0.38 for the same queries. Both methods produce salient embeddings of novel users, but IUR retains an edge over the baseline. Varying episode length. For both the proposed IUR and our extension of \newciteshrestha2017convolutional, performance increases as episode length increases. Furthermore, even for the shortest episodes considered, the proposed approach performs better. This illustrates that the choice of episode length should be decided on an application-specific basis. For example, for social media moderation, it may be desirable to quickly identify problematic users on the basis of as few posts as possible.