Checkpointing and resuming¶
Save checkpoints during a run, then continue from a previous job:
job = client.finetune(
model="qwen2.5-0.5b",
training_mode="low",
data=data,
time_limit="08:00:00",
checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)
if result["status"] != "completed":
resumed = client.finetune(
model="qwen2.5-0.5b",
training_mode="low",
data=data,
resume_from=job, # or a job ID
)
resumed.run(wait=True)
Output layout¶
| Path | Contents |
|---|---|
output/checkpoints/checkpoint-N/ |
Snapshots saved during training |
output/final/ |
Final weights |
List checkpoints with client.list_checkpoints(job_id). See dataspires.docs("checkpoints").