Self-Attentional Models for Lattice Inputs
Figure 5: A sequential lattice, and a variant with a duplicated path, where nodes a and a’ are labeled with the same word token. The matrices contain pairwise reaching probabilities in forward direction, where rows are queries, columns are keys.
['sequential', 'S', 'a', 'E']
but with a duplicated path. Semantically, both are equivalent, and should therefore result in identical neural representations. Note that while in practice duplicated paths should not occur, paths with partial overlap are quite frequent. It is therefore instructive to consider this hypothetical situation. and v{a} to abbreviate v(x{a}). Let us consider the computed representation for the node S as query. For the sequential lattice with binary mask, it is: (11) Here, C is the softmax normalization term that ensures that exponentiated similarities sum up to 1.