Skip to content

Working with your model

The steps below apply to finetune() outputs. Fine-tuning produces a LoRA adapter, which is loaded on top of the base model.

Convert to GGUF for Ollama or llama.cpp

Merge the adapter into the base model first:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")

Then convert ./my-model-merged with llama.cpp's convert_hf_to_gguf.py.

Publish to Hugging Face Hub

from huggingface_hub import HfApi

api = HfApi()

# Publish the adapter
api.upload_folder(folder_path="./my-model", repo_id="you/my-adapter", repo_type="model")

# Or publish full merged weights
api.upload_folder(folder_path="./my-model-merged", repo_id="you/my-model", repo_type="model")