Verifiable Source Code Documentation inControlled Natural Language
Table 1: The p-values for the differences between the three languages, using Wilcoxon signed rank tests.
['[EMPTY]', 'ACE∼SOUL', 'ACE∼Prolog', 'SOUL∼Prolog']
To check the observed differences between the languages for statistical significance, we use the Wilcoxon signed rank test. This is a non-parametric test to check whether or not we can conclude from observed differences within a paired sample that the difference exists in reality. This test method is robust in the sense that it does not require the statistical population to be normally distributed. These p-values represent the probability that the observed differences result by chance from a population where no differences are present. The lower these values, the more confident we can be that a difference exists in reality. All differences involving ACE are statistically significant on a 95% confidence level (and would even be significant on a 99.9% level). The differences between SOUL and Prolog are not significant.