A Read-Write Memory Network for Movie Story Understanding
Table 4: Performance of the RWMN on the video+subtitle task, according to the structure parameters of write/read networks. νw/r: the number of layers for write/read networks, (fw/rvi,sw/rvi,fw/rci): the height and the stride of convolution filters, and the number of output channels.
['# Layers  [ITALIC] νw', '# Layers  [ITALIC] νr', 'Write network ( [ITALIC] fwvi, [ITALIC] swvi, [ITALIC] fwci)', 'Read network ( [ITALIC] frvi, [ITALIC] srvi, [ITALIC] fwri)', 'Acc.']
We make several observations from the results. First, as the number of CNN layers in read /write network increases, the capacity of memory interaction may increase as well; yet the performance becomes worsen. Presumably, the main reason may be overfitting due to a relative small dataset size of MovieQA as discussed. It is hinted by our results that the two-layer CNN is the best for training performance, while the one-layer CNN is the best for validation. Second, we observe that there is no absolute magic number of how many memory slots should be read/written as a single chunk and how many strides the memory controller moves. If the stride height is too small or too large compared to the height of a convolution filter, the performance decreases. It means that the performance can be degraded when too much information is read/written as a single abstracted slot, when too much information is overlapped in adjacent reads/writes (due to a small stride), or when the information overlap is too coarse (due to a high stride). We present more ablation results to the supplementary file.