Skip to content

Checkpointing and resuming

Save checkpoints during a run, then continue from a previous job:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    time_limit="08:00:00",
    checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)

if result["status"] != "completed":
    resumed = client.finetune(
        model="qwen2.5-0.5b",
        training_mode="low",
        data=data,
        resume_from=job,          # or a job ID
    )
    resumed.run(wait=True)

Output layout

Path Contents
output/checkpoints/checkpoint-N/ Snapshots saved during training
output/final/ Final weights

List checkpoints with client.list_checkpoints(job_id). See dataspires.docs("checkpoints").