Benchmarking Popular Classification Models’ Robustness to Random and Targeted Corruptions
Table 5: Test Set accuracies after deleting the top word in the LIME explanation for different values of num_samples in comparison with the baseline accuracy
['[BOLD] Dataset', '[BOLD] Baseline  [BOLD] Accuracy', '[BOLD] Sample Size (num_samples)  [BOLD] 750', '[BOLD] Sample Size (num_samples)  [BOLD] 1500', '[BOLD] Sample Size (num_samples)  [BOLD] 3000', '[BOLD] Sample Size (num_samples)  [BOLD] 5000']
We examine 4 values of the num_samples parameter: 750, 1500, 3000, 5000 using only the FastText model. For each dataset, we construct a sample set consisting for randomly selected 10% of the documents. In the case of SST2 dataset, we use 50% of the documents since it has the lowest test data size among the datasets by far. For each value of num_samples, we identify the most important word according to LIME in each document of the four sample sets. We generate corresponding test sets where the most important word in the document is deleted. We compute the accuracies of these new test sets with the baseline FastText model. For each dataset, we look for the value of num_samples that produces the highest drop in accuracy.