Schema-Guided Dialogue State Tracking Task at DSTC8
Table 3: The best submission from each team, ordered by the joint goal accuracy on the test set. Teams marked with * submitted their papers to the workshop. We could not identify the teams for three of the submitted papers.
['[BOLD] Team name', '[BOLD] All services  [BOLD] Joint GA', '[BOLD] All services  [BOLD] Avg GA', '[BOLD] All services  [BOLD] Intent Acc', '[BOLD] All services  [BOLD] Req Slot F1', '[BOLD] Seen services  [BOLD] Joint GA', '[BOLD] Seen services  [BOLD] Avg GA', '[BOLD] Seen services  [BOLD] Intent Acc', '[BOLD] Seen services  [BOLD] Req Slot F1', '[BOLD] Unseen services  [BOLD] Joint GA', '[BOLD] Unseen services  [BOLD] Avg GA', '[BOLD] Unseen services  [BOLD] Intent Acc', '[BOLD] Unseen services  [BOLD] Req Slot F1']
The test set contains a total of 21 services, among which 6 services are also present in the training set ( seen services), whereas the remaining 15 are not present in the training set (unseen services). It also lists the performance of different submissions on seen and unseen services, helping evaluate the effectiveness in zero-shot settings. Team 9 achieved a very high joint goal accuracy of 86.53%, around 9% higher than the second-placed team. We observed the following trends across submissions: