Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
Table 2: The number of attention heads by discourse-semantic type. ‘No’ means that no linguistic annotation types are provided (attends to all tokens).
['Config', 'DR-E', 'DR-NE', 'SRL', 'Coref', 'No']
In the evaluation of multiple model configurations we notice that in some cases a single discourse/semantic type (e.g. DR (Exp)) performs better than in combination with others (e.g. SRL+DR (Exp)). Currently, we place multiple annotations on the same Encoder Block which reduces the number of free attention heads. For instance, for SRL+DR (Exp), each knowledge-enhanced encoder block has 3 SRL + 2 DR (Exp) + 3 No heads. In future work we plan to use different annotation heads per Encoder Block (EB): e.g., EB0 has 3 SRL + 5 No; EB1 has 2 DR (Exp) + 6 No; etc.