Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowledge
Table 3: Results of all sparse (S) and dense (D) models on 2 vs. 2 tests against the fMRI and MEG neuroimaging data, averaged over participants.
['[EMPTY]', '[BOLD] GloVe', '[BOLD] Word2Vec', '[BOLD] CNN-Max', '[BOLD] CNN-Mean', '[BOLD] CNN-Max + GloVe', '[BOLD] CNN-Mean + GloVe', '[BOLD] CNN-Max + Word2Vec', '[BOLD] CNN-Mean + Word2Vec']
The first step is to transform the embedding matrices and the brain activation data into a format that more readily facilitates comparison of these two very different kinds of data. For each distributional model, we calculated the pairwise correlation between concepts to produce the 60×60 similarity matrix M where each element Mi,j in the matrix is the correlation between the embedding vectors of the distributional model for the i-th and j-th concepts. In Brainbench, the brain data is already preprocessed and transformed into such a representation for both the fMRI and MEG neuroimaging modalities, giving a 60×60 similarity matrix for each participant for both modalities. The next step for BrainBench evaluation is to perform a “2 vs. 2” test between each distributional model and the brain data. Let MD and MB be the similarity matrices associated with a distributional semantic model and a participant’s brain data respectively. Let r be the Pearson correlation function, then a 2 vs. 2 test is a positive case for any two pairs of concepts w1 and w2 if r(MD(w1),MB(w1))+r(MD(w2),MB(w2)) >r(MD(w1),MB(w2))+r(MD(w2),MB(w1)) where MD(w1) and MD(w2) denote the rows of values corresponding to the concepts w1 and w2 respectively, omitting the columns that correspond to the correlation between w1 and w2. This 2 vs. 2 test is performed on all pairs of the 60 concepts, to obtain the proportion of positive cases for the pair MD and MB. The distributional models are evaluated against all brain-based representations and averaged by imaging modality. For the fMRI data, the model with the highest average 2 vs. 2 test score is the sparse multimodal GloVe+CNN-Max embedding, whilst on the MEG data the highest scoring model is a tie between the dense multimodal GloVe+CNN-Max embedding and the dense multimodal GloVe+CNN-Mean embedding. The results demonstrate that semantic distributional models that encode a range of different information are better at making statistically significant predictions on brain data. In general, the multimodal models do better than the unimodal text and image models at fitting the brain data.