BillSum: A Corpus for Automatic Summarization of US Legislation
(a) Congressional Bills
['[EMPTY]', 'Rouge-1', 'Rouge-2', 'Rouge-L']
To estimate the upper bound on our approach, an oracle summarizer is created by using the true Rouge-2 Precision scores with the MMR selection strategy. In addition, we evaluate the following unsupervised baselines: SumBasic Nenkova and Vanderwende The Rouge F-Score is used because it considers both the completeness and conciseness of the summary method. Next, we applied our US model to CA bills. Interestingly, the SUM model performs similarly to the DOC in the CA dataset, suggesting that the BERT model may have overfit to the US language. An additional reason for the similar performance is the difference in the structure of the summaries: In California the provided summaries state not only the proposed changes, but the relevant pieces of the existing law, as well (see for a more in-depth discussion). We hypothesize that a model trained on multi-state data would transfer better, thus we plan to expand the dataset to include all twenty-three states with human-written summaries.