A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
Table 1: Main Results. A * denotes a significant improvement in the F1 score over the the previous line. We compute the significance using stratified shuffling and consider it significant if the p-value <0.05.
['[EMPTY]', '[BOLD] Dataset  [BOLD] Training', '[BOLD] Dataset  [BOLD] Testing', '[BOLD] Model', '[BOLD] English  [BOLD] PU', '[BOLD] English  [BOLD] CO', '[BOLD] English  [BOLD] F1', '[BOLD] German  [BOLD] PU', '[BOLD] German  [BOLD] CO', '[BOLD] German  [BOLD] F1']
Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline. Following \newcitegarg2012unsupervised, we set the number of PRs to 2 (excluding START, END and PRED), and SRs to 21-2=19. We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets. We train only on the EP dataset and test on the CoNLL dataset. The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3. We make the following observations: [leftmargin=*] In both languages, at around S=10, the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure. Note that 10% amounts to about 3.6k sentences in German and 4k in English. We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences. At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English. This explains to some extent why only 10% labeled sentences are enough. There is no noticeable performance difference in English. We also evaluated the performance variation on a completely unseen CoNLL test set. Since the test set is very small compared to the training set, the clustering evaluation is not as reliable. Nonetheless, we broadly obtained the same pattern.