Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Table 3: Comparison of normalizing query and key in N-SAN.
['Query', 'Key', 'B@4', 'M', 'R', 'C', 'S']
What if we normalize the keys in addition to the queries? We have the following observations. 1) Normalizing either of Q and K could increase the performance. 2) The performances of normalizing both Q and K and normalizing Q alone are very similar, and are both significantly higher than that of SAN. 3) Normalizing K alone is inferior to normalizing Q alone. The reason is that normalizing K is equivalent to normalizing Θ in Eqn.