Tabula nearly rasa: Probing the linguistic knowledge of character-level neural language models trained on unsegmented text
Table 4: Italian agreement results. Random baseline accuracy is 50% in all three experiments.
['[EMPTY]', 'CNLM  [ITALIC] LSTM', 'CNLM  [ITALIC] RNN', 'WordNLM']
WordNLM shows the strongest performance, closely followed by the LSTM CNLM. The RNN CNLM performs strongly above chance (50%), but again lags behind the LSTM. where we used the adverbs più ‘more’, meno ‘less’, tanto ‘so much’. We considered only adjectives that occurred 1K times in the training corpus (as adjectives ending in -a/-o are very common). We excluded all cases in which the adverb-adjective combination occurred in the training corpus, obtaining 88 stimulus pairs. Due to the restriction to common adjectives, there were no WordNLM OOVs. Possibly, the task is made easy by the use of extremely common adverbs and adjectives. Stimulus selection was as in the last experiment, but we used a 500-occurrences threshold for adjectives, as feminine plurals are less common, obtaining 99 pairs. Again, no adverb-adjective combination was attested. There were no OOV items for the WordNLM.