O1 Preview (Sept 2024)
An evaluated model label, preserved from the source. Use the evidence below to understand what was measured and what remains unknown.
Original HumanEval tests, as recorded by EvalPlus. Less rigorous test coverage than HumanEval+.
Python function generation tested against EvalPlus expanded tests. Higher pass@1 is better.
EvalPlus sanitized MBPP subset with base tests; not the full original MBPP dataset.
Python programming problems tested against expanded EvalPlus tests. Higher pass@1 is better.
Evaluation record
- Source
- evalplus
- Publisher / namespace
- Not specified
- Prompt mode
- Instruction prompted
- Source-reported parameters
- Not provided
- Model release date
- Not provided
- Evaluation date
- Not supplied per model
- Source updated
- 26 Dec 2024
- Source revision
- 87418456d3a8c2bc6265f24ff91d2dd0e7993f4b
- Reasoning effort / harness
- Not supplied per model in this source
- Confidence intervals
- Not supplied; no significance claim
- Model license / context limit
- Not verified in this dataset
Price, speed & practical use
Verified pricing is not included in this source. We do not infer current API pricing from historical model names.
Output speed: not measured in this dataset.
Python function-completion results do not establish React, Next.js, tool-calling, or production-agent reliability. Validate candidates on your own workload.
EvalPlus contributors — results redistributed under Apache-2.0. Normalized by BuildnScale; no evaluations rerun.
Published evidence. Clear sources. No sponsored rank placement.
How to read these results →