File size: 2,130 Bytes
1c37ff9 11d2d77 0c6fcc4 51a618d 0c6fcc4 1c37ff9 11d2d77 0c6fcc4 11d2d77 0c6fcc4 11d2d77 0c6fcc4 11d2d77 0c6fcc4 f3d0388 0c6fcc4 f3d0388 0c6fcc4 f3d0388 0c6fcc4 f3d0388 0c6fcc4 f3d0388 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
---
library_name: peft
license: apache-2.0
datasets:
- maastrichtlawtech/lleqa
language:
- fr
metrics:
- rouge
- meteor
pipeline_tag: text-generation
inference: false
tags:
- legal
---
# wizardLM-7b-v1.0-lleqa
This is a [wizardlm-7b-v1.0](https://huggingface.co/WizardLM/WizardLM-7B-V1.0) model fine-tuned with [QLoRA](https://github.com/artidoro/qlora) for long-form legal question answering in **French**.
## Usage
```python
[...]
```
## Training
#### Data
We use the [Long-form Legal Question Answering (LLeQA)](https://huggingface.co/datasets/maastrichtlawtech/lleqa) dataset to fine-tune the model. LLeQA is a French native dataset for studying legal information retrieval and question answering. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.
#### Hyperparameters
We fine-tune the model through 4-bit QLoRA finetuning with an effective batch size of 8 for 10 epochs (i.e., 1.1K steps) using paged AdamW optimizer with default momentum parameters and constant learning rate schedule of 2e-4. We employ NormalFloat4 with double quantization for the base models and add LoRA adapters on all linear layers by setting r=16, alpha=32 while utilizing float16 as computation datatype. Additionally, we perform NTK-aware scaling of RoPE to extend the context window to 4096 tokens. Training takes around 7.5 hours to complete on a single Tesla V100 GPU with 32GBs of memory. More details can be found in this [paper](https://arxiv.org/abs/) and [repository](https://github.com/maastrichtlawtech/lleqa).
## Citation
```bibtex
@article{louis2023interpretable,
author = {Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos},
title = {Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models},
journal = {CoRR},
volume = {abs/2309.17050},
year = {2023},
url = {https://arxiv.org/abs/2309.17050},
eprinttype = {arXiv},
eprint = {2309.17050},
}
```
|