Working with your model¶
The steps below apply to finetune() outputs. Fine-tuning produces a LoRA adapter,
which is loaded on top of the base model.
Convert to GGUF for Ollama or llama.cpp¶
Merge the adapter into the base model first:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")
Then convert ./my-model-merged with
llama.cpp's convert_hf_to_gguf.py.