Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
Table 1: Summary of our evaluations of Calendar.help.
['[EMPTY]', '[BOLD] Study 1', '[BOLD] Study 2', '[BOLD] Study 3a', '[BOLD] Study 3b']
Each study assessed different aspects of the system at different levels of maturity. As we learned from our users’ experiences we made iterative improvements. Over the course of the three studies, Calendar.help was used by over one hundred people to schedule thousands of meetings. Starting in Study 3b, we introduced a thorough instrumentation system to measure and record detailed real-time system events and usage metrics. For this reason, we focus the analysis of usage statistics presented below on Study 3b. During this time, 178 participants delegated 1,894 meetings to their Calendar.help assistant. These involved a total of 1,981 unique meeting invitees, and 15,659 emails sent and received by Cal. Most of these meeting requests (82%, or 1,626) successfully ended in an meeting being scheduled. The remaining 268 requests that did not end in a scheduled meeting were, for one reason or another, cancelled by the organizer prior to completion. These were typically “test” requests or recruiters reaching out to people who might have not been interested in a new job.