Models and data sources¶
Pass a registry name, a Hugging Face ID, a URL, a Google Drive file, or local data. Remote sources are downloaded directly by the job, so large datasets never pass through your notebook. Local files and DataFrames are uploaded for you.
model=¶
| Input | Example |
|---|---|
| Registry name | qwen2.5-0.5b, llama-3.2-1b |
| Hugging Face ID (optional revision) | Qwen/Qwen2.5-0.5B, Qwen/Qwen2.5-0.5B@main |
| Hugging Face URL | https://huggingface.co/Qwen/Qwen2.5-0.5B |
| Google Drive file | gdrive:FILE_ID or a Drive share link |
data=¶
| Input | Example |
|---|---|
| In-memory | Python list, pandas DataFrame, Hugging Face Dataset |
| Local file or folder | ./train.jsonl, ./corpus/ |
| Hugging Face dataset | tatsu-lab/alpaca |
| Dataset with config | Salesforce/wikitext#wikitext-2-raw-v1 |
| HTTPS file | https://example.com/train.jsonl (jsonl, csv, parquet, txt, zip) |
| Google Drive file | gdrive:FILE_ID or a Drive share link |
tokenizer= (transformer pretraining)¶
Accepts gpt2, any Hugging Face tokenizer ID (optionally with @revision), a Hugging Face
URL, or a Google Drive file containing tokenizer files.
Preview how a source resolves¶
from dataspires import resolve_model, resolve_data, resolve_tokenizer
resolve_model(model="Qwen/Qwen2.5-0.5B")
resolve_data(data="Salesforce/wikitext#wikitext-2-raw-v1")
resolve_tokenizer("gpt2")
Dataset format (finetune)¶
A single text column works best. These common layouts are converted automatically:
| Columns | Converted to |
|---|---|
messages (chat format) |
text |
user + final (optional analysis) |
Instruction-style text |
instruction + output, or prompt + completion |
Instruction-style text |
Google Drive¶
Pass a Drive file the same way as any other source:
- Shared files ("Anyone with the link") work immediately.
- Private files open a Google sign-in window the first time. After you approve read-only access, the job continues and future jobs reuse your sign-in.
For headless environments, set GOOGLE_DRIVE_TOKEN instead. To turn off the sign-in
prompt, set DATASPIRES_GDRIVE_AUTO_CONNECT=0. To sign out, call
client.disconnect_google_drive().
Check a file before submitting a job:
Google Drive folders are not supported directly — share a .zip of the folder. See
Current limitations for the full list.