Read the metric
Python programming problems tested against expanded EvalPlus tests. Higher pass@1 is better. The benchmark has 399 tasks in the EvalPlus subset. A result refers to the evaluated model and prompting setup. Our initial historical snapshot preserves the source-reported pass@1 percentage.
Why expanded tests matter
A generated function can pass a small set of examples and still fail edge cases. EvalPlus adds tests to detect more incorrect solutions. Select the expanded and base metrics separately to inspect the difference; we do not average them into an overall score.
What this does not measure
These Python tasks do not establish React component quality, repository-level issue resolution, deployment correctness, or agent tool-use reliability. Published static tasks can be affected by contamination. Historical scores do not establish current API availability.
From benchmark to your application
Shortlist candidates using relevant evidence, then run representative tests from your own application. Record exact versions, prompts, tools, retries, costs, and failures. For small studies, report uncertainty and avoid definitive winner claims.