Gender and Interest Targeting for Sponsored Post Advertising at Tumblr
Table 9: Editorial evaluation of random user predictions
['[BOLD] Class Prediction', '[BOLD] Correct', '[BOLD] Wrong', '[BOLD] Not sure']
In addition to evaluation on the hold-out set, we also editorially evaluated gender predictions on the unlabeled data set of user profiles. We randomly picked 1,007 gender predictions from the population of 64.1 million users and asked editors to visit their profiles and verify the gender. They were instructed to mark our predictions as “correct", “incorrect", or “not sure". The “not sure" grade is to be used when visual inspection of profile is not conclusive, as we found was often the case. The editorial judgment came back with 573 “correct" (429 females and 144 males) The fact that there are so many “not sure" grades indicates that in many cases it is hard to infer the gender even after manual efforts, further indicating the benefits of the proposed approach and its superior performance when compared to humans. Finally, we retrained the models with 100% of golden set and deployed it in Yahoo production systems.