Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Table 3: Situation train, dev and test split for 0shot-tc.
['[EMPTY]', '[EMPTY]', 'situations search', 'situations evac', 'situations infra', 'situations utils', 'situations water', 'situations shelter', 'situations med', 'situations food', 'situations reg.', 'situations terr.', 'situations crim.', 'situations none']
To better understand the impact of generated hypotheses, we dive into the performance of each labels, taking “situation detection” as an example. This enables us to check in detail how easily the constructed hypotheses can be understood by the entailment model. Unfortunately, some classes are still challenging, such as “evacuation”, “infrastructure”, and “regime change”. This should be attributed to their over-abstract meaning. Some classes were well recognized, such as “water”, “shelter”, and “food”.