Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Table 2: Comparison of using various normalization methods in NSA.
['Approach', 'B@4', 'M', 'R', 'C', 'S']
Since we introduced IN into the NSA module for normalization, an intuitive question to ask is whether we can replace IN with other normalization methods. We have the following observations. 1) Using LN slightly decreases the performance. We conjecture that is because LN normalizes activations of all channels with the same normalization terms (μ and σ), thus limiting the expression capacity of each channel when calculating attention weights. 2) IN and IN w/o γ,β significantly outperform SAN and all the other normalization methods. Meanwhile, the extra affine transformations (γ and β) are not necessary. 3) Applying BN outperforms SAN but is inferior to adopting IN. BN has a similar effect as IN to reduce the internal covariate shift by fixing the distribution of the queries. However, as is described in Sec. .