BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding
Table 1: Pre-trained vs Fine-tuned BERT with no end task Fine Tuning
['Metric (weighted)', 'BERT', 'FT BERT']
When training our BERT-based models, we also fine tuned the BERT layers on the end task. In order to assess the delta from using the Language Model (LM) that was fine-tuned on our legal corpus, we performed another experiment where we froze the BERT layers and only trained the last portion of the network. While the final accuracy of this model was sub par, even compared to our baseline model, the gains from using a fine-tuned instead of a pre-trained LM is much more pronounced, providing further evidence for the value of domain-specific fine tuning.