Deep Multimodal Neural Architecture Search
Search Space: Per-type accuracies of MMnasNet with different decoder operation pools. All models use the same encoder operation pool of {SA,FFN}.
['Decoder operations', 'All', 'Y/N', 'Num', 'Other']
We run a number of ablations experiments on VQA-v2 to analyze the reason behind MMnasNet’s effectiveness. Search Space: Model Depth: The results reveal that: 1) MMnasNet consistently outperforms MCAN, especially when the model depth is relatively shallow (e.g., M≤8). This can be explained that the optimal architectures for different model depths are quite different; 2) with the same M and N, the model size of MMnasNet is slightly larger than MCAN. This is because MMnasNet tends to use more FFN operations, which introduces more parameters to increase the nonlinearity of the model; and 3) with the increase of model depth, both MCAN and MMnasNet saturate at M=12 and N=18, which reflects the bottleneck of the used deep encoder-decoder framework. Random vs. Searched: This suggests the used primitive operations that constitute the architecture also play a key role in model performance.