Why do you say they are similar? Interpretable Semantic Textual Similarity
Table 11: Second user study: Agreement level with grade [0..10] with non verbalized items, gold standard verbalized items and system verbalized items.
['[EMPTY]', 'No verb', 'SYS verb', 'GS verb']
In the second user study the results correspond to the agreement level in each scenario (cf. The table reports both the mean agreement level (the average of the raw agreement level introduced by the user), and the binary agreement (how many times the user entered an agreement of 5 or larger). In this user study, the effect of system verbalizations is not as clear as in the previous case: the binary agreement is better (83% vs. 77%) but the mean agreement level is very similar (7.6 vs. 7.4 agreement). The automatic verbalizations produced using gold standard annotations do have a clear impact in the task (94% vs. 77% binary agreement, and 8.8 vs. 7.4 agreement level), as the users tend to agree more with the scores assigned by the lecturer.