README.md

metadata

base_model:
  - unsloth/Meta-Llama-3.1-8B-bnb-4bit
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - llama
  - trl
license: apache-2.0
language:
  - en
datasets:
  - CineAI/Free_Thought_Frontiers
pipeline_tag: text-generation

Training configs

1. LoRA Rank: 16
2. Max sequence length: 2048
3. Max steps: 60
4. Learning rate: 2e-4

How to use model

You can use the following example using the Unsloth interface

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""


from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "CineAI/Free-Thought-Frontiers-Llama32-8B",
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference

inputs = tokenizer(
[
    alpaca_prompt.format(
        "Insert here quote or ststement", # instruction
        "", # input leave empty
        "", # output - leave this blank for generation!
    )
], return_tensors = "pt").to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 2048)

Or you can use AutoModelForPeftCausalLM, but it is 2x slower than Unsloth

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
model = AutoPeftModelForCausalLM.from_pretrained(
    "CineAI/Free-Thought-Frontiers-Llama32-8B",
    load_in_4bit = True,
)
tokenizer = AutoTokenizer.from_pretrained("CineAI/Free-Thought-Frontiers-Llama32-8B")

Uploaded model

• Developed by: CineAI
• License: apache-2.0
• Finetuned from model : unsloth/meta-llama-3.1-8b-bnb-4bit

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.