A Critical Analysis of What is Learned in Visually Grounded Neural Syntax Acquisition
Table 1: Test results. We report the results from Shi2019:vgsnl as Shi2019 and our reproduction (Shi2019∗). We report mean F1 and standard deviation for each system and recall for four phrasal categories. Our variants are specified using a representation embedding (d∈{1,2}), a score function (sM: mean, sMHI: mean+HI, sWS: weighted sum), and a combine function (cMX: max, cME: mean).
['Model', 'NP', 'VP', 'PP', 'ADJP', 'Avg.  [ITALIC] F1']
As the table illustrates, The model variations achieve F1 scores competitive to the scores reported by Shi2019:vgsnl across training setups. The model variations closest to the original model, 1,sWS,cME and 2,sWS,cME, yield similar performance to the original model across different evaluation categories and metrics, especially in the +HI and +HI+FastText settings. Most remarkably, our simplest variants, which use 1d embeddings and a non-parameterized scoring function, are still competitive (1,sM,cME) or even outperform (1,sMHI,cMX) the original VG-NSL. We include standard deviation for the phrasal category recall and self-F1 scores evaluated across different parameter initializations. It visualizes the token embeddings of 1,sWS,cME and 2,sWS,cME for all universal parts-of-speech categories (Petrov:12universalpos). Figures LABEL:fig:concreteness_high and LABEL: fig: concreteness_low show several examples visualizing our learned representations with the 1,sWS,cME variant, the 1d variant closest to the original model, as a concreteness estimate. Figure LABEL:fig: concreteness_high shows the most concrete nouns, and Figure LABEL:fig: concreteness_low shows the least concrete nouns. We selected nouns from the top (bottom) 5% of the data as most (least) concrete. We randomly selected image-caption pairs for these nouns.