Message Passing Attention Networks for Document Understanding
Table 3: Ablation results. The n in nMP refers to the number of message passing iterations. *vanilla model (MPAD in Table 2).
['[BOLD] MPAD variant', 'Reut.', 'Pol.', 'IMDB']
To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD. Number of MP iterations. First, we varied the number of message passing iterations from 1 to 4. We attribute this to the fact that we are reading out at each iteration from 1 to T (see Eq . Indeed, in initial experiments involving readout at t= T only, setting T≥2 was always decreasing performance, despite the GRU-based updates ( Eq. These results were consistent with that of [yao2019graph] and [kipf2016semi], who both are reading out only at t =T too. We hypothesize that node features at T≥2 are too diffuse to be entirely relied upon during readout. More precisely, initially at t=0, node representations capture information about words, at t=1, about their 1-hop neighborhood (bigrams), at t=2, about compositions of bigrams, etc. Thus, pretty quickly, node features become general and diffuse. In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary. No renormalization. That is, Eq. Unlike the mean, which captures distributions, the sum captures structural information [xu2018powerful]. We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc. of the corresponding documents.