RSDD-Time: Temporal Annotation of Self-Reported Mental Health Diagnoses
Table 4: Results using SUTime, with additional rules for predicting age expressions and when limiting the candidate expression set using the closest heuristic.
['[EMPTY]', '[BOLD] P', '[BOLD] R', '[BOLD] F1']
Because diagnoses often include an age or year in school rather than an absolute time, we added rules specifically to capture these time expressions. The rules were manually generated by examining the training data, and will be released alongside the annotations. RSDD-Time temporal expression annotations are only concerned with time expressions that relate to the diagnosis, whereas SUTime extracts all temporal expressions in a given text. We use a simple heuristic to resolve this issue: simply choose the time expression closest to the post ’s diagnosis by character distance. In the case of a tie, the heuristic arbitrarily selects the leftmost expression. This heuristic will improve precision by eliminating many unnecessary temporal expressions, but has the potential to reduce precision by eliminating some correct expressions that are not the closest to the diagnosis. Notice that custom age rules greatly improve the recall of the system. The experiment also shows that the closest heuristic improves precision at the expense of recall (both with and without the age rules). Overall, the best results in terms of F1 score are achieved using both the closest heuristic and the age rules. A more sophisticated algorithm could be developed to increase the candidate expression set (to improve recall), and better predict which temporal expressions likely correspond to the diagnosis (to improve precision).