Skip to content

Evaluate

Use client.evaluate() to score a text model on a benchmark, or on questions you write. The weights do not change. Evaluation runs as a GPU job, priced like any other job (see Hardware and billing).

job = client.evaluate(
    model="Qwen/Qwen2.5-0.5B",
    tasks="gsm8k",
    max_samples=20,
)
result = job.run(wait=True)

if result["status"] == "completed":
    print(job.get_results())
    job.export_results("./eval_results.json")

tasks="gsm8k" scores grade-school math with LightEval. max_samples=20 scores 20 questions instead of the full set, which is quicker and cheaper while you try things out.

Benchmarks

Pass one task name, or a list to score several in the same job:

job = client.evaluate(
    model="Qwen/Qwen2.5-0.5B",
    tasks=["gsm8k", "hellaswag"],
    few_shot=5,
    max_samples=20,
)
Argument Meaning
tasks LightEval task names, for example "gsm8k" or "hellaswag".
few_shot How many answered examples the model sees before each question. Default 0.
max_samples Score only this many questions per task. Default: the full set.

To set the number of examples for one task only, add it after the task name, for example "gsm8k|5".

Your own questions

Each row needs a question and the answer you expect. A reply counts when it matches the answer or starts with it, ignoring case and extra spaces.

job = client.evaluate(
    model="Qwen/Qwen2.5-0.5B",
    data=[
        {"question": "The capital of France is", "answer": "Paris"},
        {"question": "2 + 2 =", "answer": "4"},
    ],
)
result = job.run(wait=True)

data can also be a JSONL file (data="./exam.jsonl") or a pandas DataFrame. prompt works in place of question, and target in place of answer. The SDK checks every row before it asks for a GPU, so a bad row fails right away, at no charge.

Argument Meaning
data The exam: rows, a JSONL path, or a DataFrame
metric How replies are scored. "exact_match" (the default) is the only one today.
max_new_tokens The longest reply the model may write. Default 64.

You can pass both tasks and data; the job scores both.

A model you trained

Pass a finished text LoRA job as source_job_id. Its adapter is applied to the base model, which is then scored the same way:

train = client.finetune(model="qwen2.5-0.5b", training_mode="low", data=data)
train.run(wait=True)

job = client.evaluate(
    source_job_id=train.job_id,
    tasks="gsm8k",
    max_samples=20,
)

The training job's output must include output/final/adapter_config.json. An evaluate job, a full fine-tune, and a vision model do not have one.

evaluate() is for text models. Classifiers, detectors, and vision-language models keep the scores from training (see get_metrics() on the training job).

Results

job.get_results() returns a dict; job.export_results(path) saves the same as JSON.

{
    "model": "Qwen/Qwen2.5-0.5B",
    "source_job_id": None,
    "adapter": False,
    "tasks": [
        # One entry per task; scores are LightEval's metrics for that task.
        {"task": "gsm8k|0", "scores": {"gsm8k|0:<metric>": 0.35}},
    ],
    "custom": {
        "metric": "exact_match",
        "n": 2,
        "correct": 2,
        "score": 1.0,
        "rows": [
            {"question": "2 + 2 =", "answer": "4", "prediction": "4", "correct": True},
        ],
    },
}

custom.rows holds every question with the model's reply, so you can see which ones it got wrong. If the job fails, the file still has an error field with the reason.

GPU and precision

evaluate() takes the same GPU options as training: nothing (our V100), gpu_type=, or offering= (see Choosing a GPU). The default time_limit is "02:00:00".

dtype="auto" (the default) scores in bfloat16 on GPUs that support it and in float16 on the V100, which has no native bfloat16. Pass dtype="float32" for full precision at about twice the memory.