Skip to content

Models and data sources

Pass a registry name, a Hugging Face ID, a URL, a Google Drive file, or local data. Remote sources are downloaded directly by the job, so large datasets never pass through your notebook. Local files and DataFrames are uploaded for you.

model=

Input Example
Registry name qwen2.5-0.5b, llama-3.2-1b
Hugging Face ID (optional revision) Qwen/Qwen2.5-0.5B, Qwen/Qwen2.5-0.5B@main
Hugging Face URL https://huggingface.co/Qwen/Qwen2.5-0.5B
Google Drive file gdrive:FILE_ID or a Drive share link

data=

Input Example
In-memory Python list, pandas DataFrame, Hugging Face Dataset
Local file or folder ./train.jsonl, ./corpus/
Hugging Face dataset tatsu-lab/alpaca
Dataset with config Salesforce/wikitext#wikitext-2-raw-v1
HTTPS file https://example.com/train.jsonl (jsonl, csv, parquet, txt, zip)
Google Drive file gdrive:FILE_ID or a Drive share link

tokenizer= (transformer pretraining)

Accepts gpt2, any Hugging Face tokenizer ID (optionally with @revision), a Hugging Face URL, or a Google Drive file containing tokenizer files.

Preview how a source resolves

from dataspires import resolve_model, resolve_data, resolve_tokenizer

resolve_model(model="Qwen/Qwen2.5-0.5B")
resolve_data(data="Salesforce/wikitext#wikitext-2-raw-v1")
resolve_tokenizer("gpt2")

Dataset format (finetune)

A single text column works best. These common layouts are converted automatically:

Columns Converted to
messages (chat format) text
user + final (optional analysis) Instruction-style text
instruction + output, or prompt + completion Instruction-style text

Google Drive

Pass a Drive file the same way as any other source:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="gdrive:FILE_ID",
    gpus=1,
)
  • Shared files ("Anyone with the link") work immediately.
  • Private files open a Google sign-in window the first time. After you approve read-only access, the job continues and future jobs reuse your sign-in.

For headless environments, set GOOGLE_DRIVE_TOKEN instead. To turn off the sign-in prompt, set DATASPIRES_GDRIVE_AUTO_CONNECT=0. To sign out, call client.disconnect_google_drive().

Check a file before submitting a job:

client.inspect_source("gdrive:FILE_ID")

Google Drive folders are not supported directly — share a .zip of the folder. See Current limitations for the full list.