Lexical Bias In Essay Level Prediction
Table 1: Cost matrix used to calculate the miscalssification error described in Eq. (1).
['[EMPTY]', '[ITALIC] A1', '[ITALIC] A2', '[ITALIC] B1', '[ITALIC] B2', '[ITALIC] C1', '[ITALIC] C2']
In order to capture this explicit ordering of L, the organisers proposed a cost measure that uses the confusion matrix of the prediction and prior knowledge in order to evaluate the performance of the system. In particular, the meaures uses writes as: E=100n6∑i=16∑j=1Ci,jNi,j (1) where C is a cost matrix that uses prior knowledge to calculate the misclassification errors and Ni ,j is the number of observations of class i classified with category j. Notice that, as expected, moving away from the diagonal (correct classification) the misclassification costs are higher. The biggest error (44) occurs when a C2 essay is classified as A1. On the contrary, the classification error is lower (6) when the opposite happens and an A1 essay is classified as C2. Since C is not symmetric and the costs of the lower diagonal are higher, the penalties for misclassification are worse when essays of upper languages levels (e.g., C1,C2) are classified as essays of lower levels.