Event Representations with Tensor-based Compositions
Table 4: Average Annotator Scores for generated schemas (scale from 0-4). Higher scores indicate a more coherent schema.
['System', 'Average Score With 0’s', 'Average Score Without 0’s']
Our evaluation is human-driven. We present annotators with 3 schemas (from the 3 models) using the same seed. For each event in each schema, annotators rated the event on a 0-4 scale with regards to its relevance to the seed and the rest of the schema. A score of 0 is reserved for events that are completely nonsensical (either caused by an extraction error, or bad entity replacement). A score of 1 indicates that the event was not relevant with no obvious relation to the scenario, while a score of 4 indicates the event is highly relevant and would be a core part of any description of the scenario.