From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
Table 5: An illustrative sentence with its nearest neighbour at layer 1 and layer 3. For readability, sentences are displayed in conventional spelling, and only highlight matching substrings of length ≥3. In reality we used phonetic transcriptions to compute shared substring positions, and substrings of all lengths.
['Layer', 'Mean position']
Here we quantify the time-scale at which information is retained in the different layers of Phon GRU. We looked at the location of phoneme strings shared by sentences and their nearest neighbors in the 5,000-image validation sample. We determined each sentence’s nearest neighbor for each hidden layer in Phon GRU. The nearest neighbour is the sentence for which the activation vector at the end of sentence symbol has the smallest cosine distance to the activation vector of the original sentence. The position of matching substrings is the average position in the original sentence of symbols in substrings that are shared by the neighbor sentences, counted from the end of the sentence. A high mean average substring position thus means that the shared substring(s) appear early in the sentence. This gives an indirect measure of the timescale at which the different layers operate. This indicates a difference between the layers with regards to the timescale they represent. Whereas in the lowest layer only information from the latest timesteps is present, the higher layers retain the input signal over longer timescales.