Abstract
Table 4: Statistical similarity between different samples at different length of n-grams.
['n', 'SM', 'MW', 'SW']
(We are publishing this figure as it is the worst – under the other conditions, the match is even better.) In terms of cosine similarity, the same tendencies that we established for unigram data remain true for bigram or higher POS n-grams: the Switchboard data is quite far from both Simple and Main, the Brown Corpus is closer, and the WSJ is closest. We chose this condition because we believe it to be the least noisy, but we emphasize that the same relations are observed for all other conditions, with or without sentence boundary postprocessing, with or without removal of machine-generated entries from Simple, with or without readjusting the Main corpus to reflect this change (all 32 combinations were investigated). The data leave no doubt that the WSJ is closer to Main than to Simple, but the angles are large enough, especially when compared to the Simple/Main column, to discourage any attempt at explaining the syntax of Main, or Simple, based on the syntax of well-edited journalistic prose. We conclude that the simplicity of Simple, evident both from reading the material and from the Gunning Fog index discussed above, is due primarily to Main having considerably longer sentences. A secondary effect may be the use of shorter subsentences (comma-separated stretches) as well, but this remains unclear in that the number of subsentence separators (commas, colons, semicolons, parens, quotation marks) per sentence is considerably higher in Main (1.62) than in Simple (1.01), so a Main subsentence is on the average not much longer than a Simple subsentence (8.62 vs 7.96 content words/subsentence).