Dependency Parsing for Spoken Dialog Systems
Table 3: Parsing Accuracy Results
['[BOLD] Dataset', '[BOLD] UAS', '[BOLD] LAS']
Training only using 1,000 annotated in-domain ConvBank data reaches 75.65% and 58.29% accuracy on unlabeled and labeled edges respectively. This suggests that 1,000 utterances are not enough to achieve optimal results. However, human annotations are expensive to obtain, limiting the size of our dataset. The model trained on EWT or Tweebank or EWT and Tweebank combined achieved similar results that are better than the model trained on ConvBank alone. As these three models had similar results, we fine-tune on the slightly better-performing EWT model. Fine-tune achieves the best best result on both unlabeled and labeled edges (85.05% and 77.82% respectively). Fine-tuning on EWT with ConvBank data improves parsing accuracy on the test data by over 7 points on unlabeled dependency edges and by nearly 12 points on labeled edges. These results suggest leveraging both a large set of out-of-domain data and in-domain data achieves the best performance.