Twitter Job/Employment Corpus: A Dataset of Job-Related Discourse Built with Humans in the Loop
Table 5: Crowdsourced validations of samples identified by models C0, C1, C2 and C3.
['[BOLD] Model', '[BOLD] Class', '[BOLD] P', '[BOLD] R', '[BOLD] F1']
We proposed an end-to-end evaluation: to what degree can our trained automatic classifiers (C1, C2, C3 and C5) identify job-related tweets in the real world? We introduced the estimated effective recall under the assumption that for each model, the error rates in our test samples (1,600 tweets) are proportional to the actual error rates found in the entire one-year data set which resembles the real world. We labeled the entire data set using each classifier and defined the estimated effective recall ^R for each classifier as ^R = Y⋅Nt⋅RY⋅Nt⋅R+N⋅Yt⋅(1−R)