Factorized Multimodal Transformer for Multimodal Sequential Learning
Table 7: Results of experiments with different number of heads for OTF. The number in the square bracket indicates the number of heads.
['[ITALIC] Model \\ [ITALIC] Metric', '[ITALIC] BA', '[ITALIC] F1', '[ITALIC] MAE', '[ITALIC] Corr']
In this section, we discuss the effect of increasing the number of heads on the original transformer model (OTF, Vaswani et al. Please note that we implement the OTF to allow for all attention heads to have full input receptive field (from 1…T), similar to FMT. We increase the attention heads from 1 to 35 (after 35 does not fit on a Tesla-V100 GPU with batchsize of 20). We observe that achieving superior performance is not a matter of increasing the attention heads. Even using 1 FMS unit, which leads to 7 total attention, FMT achieves higher performance than counterpart OTF.