Evaluate¶
Use client.evaluate() to score a text model on a benchmark, or on questions you write.
The weights do not change. Evaluation runs as a GPU job, priced like any other job (see
Hardware and billing).
job = client.evaluate(
model="Qwen/Qwen2.5-0.5B",
tasks="gsm8k",
max_samples=20,
)
result = job.run(wait=True)
if result["status"] == "completed":
print(job.get_results())
job.export_results("./eval_results.json")
tasks="gsm8k" scores grade-school math with LightEval.
max_samples=20 scores 20 questions instead of the full set, which is quicker and cheaper
while you try things out.
Benchmarks¶
Pass one task name, or a list to score several in the same job:
job = client.evaluate(
model="Qwen/Qwen2.5-0.5B",
tasks=["gsm8k", "hellaswag"],
few_shot=5,
max_samples=20,
)
| Argument | Meaning |
|---|---|
tasks |
LightEval task names, for example "gsm8k" or "hellaswag". |
few_shot |
How many answered examples the model sees before each question. Default 0. |
max_samples |
Score only this many questions per task. Default: the full set. |
To set the number of examples for one task only, add it after the task name, for example
"gsm8k|5".
Your own questions¶
Each row needs a question and the answer you expect. A reply counts when it matches the answer or starts with it, ignoring case and extra spaces.
job = client.evaluate(
model="Qwen/Qwen2.5-0.5B",
data=[
{"question": "The capital of France is", "answer": "Paris"},
{"question": "2 + 2 =", "answer": "4"},
],
)
result = job.run(wait=True)
data can also be a JSONL file (data="./exam.jsonl") or a pandas DataFrame. prompt
works in place of question, and target in place of answer. The SDK checks every row
before it asks for a GPU, so a bad row fails right away, at no charge.
| Argument | Meaning |
|---|---|
data |
The exam: rows, a JSONL path, or a DataFrame |
metric |
How replies are scored. "exact_match" (the default) is the only one today. |
max_new_tokens |
The longest reply the model may write. Default 64. |
You can pass both tasks and data; the job scores both.
A model you trained¶
Pass a finished text LoRA job as source_job_id. Its adapter is applied to the base model,
which is then scored the same way:
train = client.finetune(model="qwen2.5-0.5b", training_mode="low", data=data)
train.run(wait=True)
job = client.evaluate(
source_job_id=train.job_id,
tasks="gsm8k",
max_samples=20,
)
The training job's output must include output/final/adapter_config.json. An evaluate job,
a full fine-tune, and a vision model do not have one.
evaluate() is for text models. Classifiers, detectors, and vision-language models keep
the scores from training (see get_metrics() on the training job).
Results¶
job.get_results() returns a dict; job.export_results(path) saves the same as JSON.
{
"model": "Qwen/Qwen2.5-0.5B",
"source_job_id": None,
"adapter": False,
"tasks": [
# One entry per task; scores are LightEval's metrics for that task.
{"task": "gsm8k|0", "scores": {"gsm8k|0:<metric>": 0.35}},
],
"custom": {
"metric": "exact_match",
"n": 2,
"correct": 2,
"score": 1.0,
"rows": [
{"question": "2 + 2 =", "answer": "4", "prediction": "4", "correct": True},
],
},
}
custom.rows holds every question with the model's reply, so you can see which ones it got
wrong. If the job fails, the file still has an error field with the reason.
GPU and precision¶
evaluate() takes the same GPU options as training: nothing (our V100), gpu_type=, or
offering= (see Choosing a GPU).
The default time_limit is "02:00:00".
dtype="auto" (the default) scores in bfloat16 on GPUs that support it and in float16 on
the V100, which has no native bfloat16. Pass dtype="float32" for full precision at about
twice the memory.