Smart To-Do : Automatic Generation of To-Do Items from Emails
Table 2: Performance of unsupervised approaches in identifying helpful sentences for a given query.
['[BOLD] Algorithm', 'At-least One Helpful  [BOLD] @ K=2', 'At-least One Helpful  [BOLD] @ K=3']
We also fine-tuned BERT on the labeled dataset for commitment classifier. The dataset is first made balanced (2586 positive and 2586 negative instances). Uncased BERT is trained for 5 epochs for commitment classification, with the input being word-piece tokenized email sentences. We manually annotated 100 email instances and labeled every sentence as helpful or not based on (a) whether the sentence contains concepts appearing in the target To-Do item, and (b) whether the sentence helps to understand the task context. Inter-annotator agreement between 2 judgments for this task has a Cohen Kappa score of 0.69. This annotation task also demonstrates the importance of the previous email in a thread. Out of 100 annotated instances, 44 have a replied-to email of which 31 contains a helpful sentence in the replied-to email body (70.4%). FastText with max-pooling of embeddings performed the best and used in the subsequent generation stage.