Update README with KL3M tokenizer paper citation - README.md

README.md

@@ -10,6 +10,7 @@ tags:
 - financial
 - enterprise
 - slm
+- gpt-neox
 date: '2024-02-20T00:00:00.000Z'
 pipeline_tag: text-generation
 widget:
@@ -18,9 +19,9 @@ widget:
  - do_sample: True
 ---
 
-# kl3m-003-1.7b Model
+# kl3m-003-1.7b
 
-kl3m-1.7b is a small language model (SLM) model trained on clean, legally-permissible data. Originally 
+kl3m-003-1.7b is a small language model (SLM) trained on clean, legally-permissible data. Originally 
 developed by [273 Ventures](https://273ventures.com) and donated to the [ALEA Institute](https://aleainstitute.ai), 
 kl3m-003-1.7b was part of the first LLM family to obtain the [Fairly Trained L-Certification](https://www.fairlytrained.org/certifications)
 for its ethical training data and practices. The model is designed for legal, regulatory, and financial workflows,
@@ -29,40 +30,34 @@ with a focus on low toxicity and high efficiency.
 Given its small size and lack of training data for instruction alignment, kl3m-003-1.7b is best suited for use either in
 SLM fine-tuning or as part of training larger models without using unethical data or models.
 
-The model was originally trained between January-February 2024 on a 8xA100-80G node in DDP.  A similar model is
-being provided with complete source and data replication as part of the `kl3m-004` family to be released in Q4 2024.
-
-## Source
-
-[https://github.com/alea-institute/kl3m-model-research](https://github.com/alea-institute/kl3m-model-research)
-
-
-## Training Data
-While the original training data collection and training infrastructure relies on software that was not donated by
-273 Ventures, ALEA Institute is open-sourcing an improved dataset, including both replication and an API.
-
-[https://github.com/alea-institute/kl3m-data](https://github.com/alea-institute/kl3m-data)
-
-Data is available upon request at this time via S3 under a Requester Pays model.  We are actively working on a
-zero-cost distribution model as soon as we can obtain additional support.
-
-This model, the original `kl3m-003-1.7b` model, was trained on a US-only subset of the Kelvin Legal DataPack that
-we believe is 100% public domain material. However, so as to enforce maximum transparency to all 
-downstream users in the event of any future determination otherwise, we are licensing this model under CC-BY 4.0.
-
 ## Model Details
 
-### Summary
 - **Architecture**: GPT-NeoX (i.e., ~GPT-3 architecture)
-- **Parameters**: 1.7 billion
+- **Size**: 1.7 billion parameters
+- **Hidden Size**: 2048
+- **Layers**: 32
+- **Attention Heads**: 32
+- **Intermediate Size**: 8192
+- **Max Position Embeddings**: 8192
 - **Context Window**: 8,192 tokens (true size, no sliding window)
+- **Tokenizer**: [kl3m-001-32k](https://huggingface.co/alea-institute/kl3m-001-32k) BPE tokenizer (32,768 vocabulary size with unorthodox whitespace handling)
 - **Language(s)**: Primarily English
-- **Tokenizer**: kl3m-001-32k BPE tokenizer (32,768 vocabulary size with unorthodox whitespace handling)
+- **Training Objective**: Next token prediction
 - **Developed by**: Originally by [273 Ventures LLC](https://273ventures.com), donated to [ALEA Institute](https://aleainstitute.ai)
 - **License**: [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/)
 - **Hardware Requirements**: Runs real-time in bf16 on consumer NV/AMD GPUs
 
-## Performance Metrics
+## Use Cases
+
+kl3m-003-1.7b is particularly effective for:
+
+- Basic regulatory question answering
+- Contract provision drafting
+- Structured JSON information extraction
+- Foundation for downstream optimization
+- Base model for domain-specific fine-tuning
+
+## Performance
 
 ### Perplexity Scores
 | Dataset        | Score |
@@ -81,15 +76,9 @@ larger models as of its training data.
 - **Enterprise Focus**: Specifically designed for legal, regulatory, and financial workflows.
 - **Efficient Deployment**: Optimized for real-time inference on consumer hardware.
 
-## Use Cases
-
-- Basic regulatory question answering
-- Contract provision drafting
-- Structured JSON information extraction
-- Foundation for downstream optimization
-- Base model for domain-specific fine-tuning
+## Usage
 
-## Getting Started
+Basic usage for text generation:
 
 ```python
 import json
@@ -98,7 +87,7 @@ from transformers import pipeline
 # Load the model and tokenizer
 p = pipeline('text-generation', 'alea-institute/kl3m-003-1.7b', device='cuda')
 
-# Example usage on CPU
+# Example usage on GPU
 text = "Under this"
 print(
     json.dumps(
@@ -115,12 +104,12 @@ print(
 [
   "Under this section, any person who is a party to the proceeding may be required to file ",
   "Under this subsection, the term **eligible entity** means a State, a political subdivision of ",
-  "Under this section, the Secretary shall\u2014 (1)\nmake a grant to the National Academy of Sc"
+  "Under this section, the Secretary shall— (1)\nmake a grant to the National Academy of Sc"
 ]
-
 ```
 
-## Contract Example
+### Contract Example
+
 ```python
 text = "Governing Law. "
 print(
@@ -142,7 +131,21 @@ print(
 ]
 ```
 
-## Technical Implementation
+### Generation Parameters
+
+The model supports various parameters to control the generation process:
+
+- `temperature`: Controls randomness (lower = more deterministic)
+- `top_p`: Nucleus sampling parameter (lower = more focused)
+- `top_k`: Limits vocabulary selection to top k tokens
+- `max_new_tokens`: Maximum number of tokens to generate
+- `do_sample`: Whether to use sampling vs. greedy decoding
+- `num_return_sequences`: Number of different sequences to generate
+
+## Training
+
+The model was originally trained between January-February 2024 on an 8xA100-80G node in DDP. A similar model is
+being provided with complete source and data replication as part of the `kl3m-004` family to be released in Q4 2024.
 
 The model implements several techniques during training:
 
@@ -151,6 +154,77 @@ The model implements several techniques during training:
 - Randomized padding
 - Traditional fixed-attention mechanisms
 
+### Training Data
+
+While the original training data collection and training infrastructure relies on software that was not donated by
+273 Ventures, ALEA Institute is open-sourcing an improved dataset, including both replication and an API.
+
+[https://github.com/alea-institute/kl3m-data](https://github.com/alea-institute/kl3m-data)
+
+Data is available upon request at this time via S3 under a Requester Pays model. We are actively working on a
+zero-cost distribution model as soon as we can obtain additional support.
+
+This model, the original `kl3m-003-1.7b` model, was trained on a US-only subset of the Kelvin Legal DataPack that
+we believe is 100% public domain material. However, so as to enforce maximum transparency to all 
+downstream users in the event of any future determination otherwise, we are licensing this model under CC-BY 4.0.
+
+## Intended Usage
+
+This model is intended for use in:
+
+- Legal and regulatory document processing systems
+- Contract drafting assistance
+- Financial and enterprise document workflows
+- Educational contexts for learning about domain-specific language models
+- Research on efficient language models for domain-specific applications
+
+## Special Tokens
+
+kl3m-003-1.7b uses standard special tokens from the GPT-NeoX architecture.
+
+## Limitations
+
+- As a small language model (1.7B parameters), it has limited general knowledge
+- Not instruction-tuned or aligned with human preferences
+- May generate plausible-sounding but incorrect legal or regulatory text
+- Not a substitute for professional legal advice or domain expertise
+- Performance is optimized for legal and financial domains; general performance may be lower
+
+## Ethical Considerations
+
+- This model should not be used to generate legal advice without human expert review
+- The model may reflect biases present in the training data despite efforts to use clean data
+- Generated text should be reviewed by qualified professionals before use in formal legal contexts
+- While trained on ethically sourced data, users should verify outputs for accuracy and appropriateness
+
+## Source
+
+[https://github.com/alea-institute/kl3m-model-research](https://github.com/alea-institute/kl3m-model-research)
+
+## References
+
+- [KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications](https://arxiv.org/abs/2503.17247)
+- Additional tokenizer, dataset, and model publications are pending.
+
+## Citation
+
+```bibtex
+@misc{kl3m-003-1.7b,
+  author = {ALEA Institute},
+  title = {kl3m-003-1.7b: A Small Language Model for Legal and Regulatory Text},
+  year = {2024},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-003-1.7b}}
+}
+
+@article{bommarito2025kl3m,
+  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
+  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
+  journal={arXiv preprint arXiv:2503.17247},
+  year={2025}
+}
+```
+
 ## License
 
 This model was originally developed by 273 Ventures and has been donated to the ALEA Institute. 
@@ -169,14 +243,4 @@ The KL3M model family is now maintained by the [ALEA Institute](https://aleainst
 
 Special thanks to 273 Ventures for developing and donating this model to the open-source community through the Alea Institute.
 
-
-## Citation
-
-Tokenizer, dataset, and model publications are pending.
-
-## Contact
-
-For any questions, please contact [ALEA Institute](https://aleainstitute.ai) at [[email protected]](mailto:[email protected]) or
-create an issue on this repository or [GitHub](https://github.com/alea-institute/kl3m-model-research).
-
-![https://aleainstitute.ai](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)
+![logo](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)