Like a Baby: Visually Situated Neural Language Acquisition
Table 1: Generalization performance as measured by negative log likelihood (NLL) and perplexity (PPL). Lower values indicate better performance. Baseline model (L-L) trained and evaluated on linguistic data only. Full model (LV-LV) trained and evaluated on both linguistic and visual data. Blind model (LV-L) trained on both but evaluated on language only. The difference between L-L and LV-L illustrates the performance improvement. German and Spanish data are machine-translated (MT) and provide additional, but correlated, evidence. For comparison, Devlin et al. (2018) report a perplexity of 3.23 for their (broad) English test data, using the same base model we use here to define input representations.
['Model (Type)', '[BOLD] English Test-NLL', '[BOLD] English Test-PPL', '[BOLD] German MT Test-NLL', '[BOLD] German MT Test-PPL', '[BOLD] Spanish MT Test-NLL', '[BOLD] Spanish MT Test-PPL']
PPL is calculated as: PPL=exp[−(1/N)N∑i=1T∑t=1logPΘ(wt|h)] We observe that in all cases the multi-modal models outperform their respective text-only baselines. More importantly, the multi-modal models, when evaluated without the Inception-v3 representations on holdout samples, still perform better than the text-only baselines. The improvement in language generalization can be attributed to the visual context information provided during training, enriching its representations over word sequences with knowledge of actual objects and actions.