Teddy: A System for Interactive Review Analysis
Table 2: BERT classifiers’ accuracy using different schemas.
['[EMPTY]', '#attr', 'coverage(%)', 'Level-1', 'Level-2', 'Level-3']
Chao re-trained his extraction classifier and evaluated the results on his test set. Each classifier is trained on 5,000 aspect-opinion pairs by fine-tuning the uncased 12-layer BERT model [devlin2019bert] for 10 epochs and is evaluated on 1,000 labeled pairs. The three versions consist of 3, 7, and 10 attributes created as described above. Each schema also contains a special attribute “others” indicating examples that are not covered. The coverage(%) column shows the percentage of test examples that are not labeled as “others.” From increasing the number of attributes from 3 to 10, the coverage increased by 14.2%. The last 3 columns show the classifiers’ accuracy. Each column “Level-i” indicates the accuracy when evaluated at the granularity of schema-i. When a classifier is evaluated at a less fine-grained (easier) level, each predicted label is mapped to the attribute that contains the predicted label (e.g., “food-quality” to “food,” or “alcohol” to “drink”). When evaluated at a finer-grained (harder) level than the classifier’s schema, each predicted label is mapped to a single attribute (For example, when schema-1 is evaluated on Level-3, a “drink” or “facility” prediction is only correct if those are the ground-truth labels, but a “food” prediction is only correct if the ground-truth is “food-quality,” since “food” is not in the level-3 schema). Chao’s results clearly showed that as the schema became more carefully designed, the accuracy of the BERT classifier increased significantly (by 12.9% at Level-2 and by 16.9% at Level-3).