[email protected] · pass@1

The HumanEval contract.

164 hand-written Python problems, deterministic decoding, one shot. Every row lands with a dataset fingerprint, a harness fingerprint, and a permalink into the workflow that produced it.

01Benchmark

HumanEval is a Python function-completion benchmark introduced by OpenAI in Evaluating Large Language Models Trained on Code (2021). 164 hand-written problems, each with a docstring, function signature, and a suite of test cases. A model is prompted with the signature + docstring and must emit a body; pass@k is the probability that at least one of k samples passes the tests.

Gaia records HumanEval as benchmarkId: [email protected], with unit: pass@1 (k = 1, deterministic decoding). Variants (pass@10, sampled, temperature-adjusted) require a different benchmarkId or unit.

02Dataset

Source: openai/human-eval — data/HumanEval.jsonl.gz.

The canonical HumanEval dataset hash is stamped on the first ci-reproduced row that lands from .github/workflows/benchmark-humaneval-ci.yml. The unit-test fixture (scripts/benchmarks/humaneval/fixtures/mini.jsonl) carries datasetHash = 244753b2a3366bfbb271e76205fdd88e939c91705093c1a18eebd60fc8a0ebf8 and is what dogfooded the pipeline.

Any change to the dataset produces a new datasetHash and therefore a new row. Two rows referencing different HumanEval dataset revisions are not comparable and must not be aggregated.

Different fingerprint, different row. Never aggregate across hashes.

03Harness

Location: scripts/benchmarks/humaneval/run.py — ships in Sprint D W2b (#905).

The harness is required to:

1. Load the dataset from the pinned URL and compute datasetHash.
2. Compute benchmarkInputHash as SHA-256 of (dataset + prompt template + harness config).
3. Run the model under evaluation with k = 1, deterministic decoding.
4. Emit a results.json containing, at minimum:
   • score (0..1 pass@1)
   • unit: "pass@1"
   • runAt (ISO 8601 with timezone)
   • datasetHash
   • benchmarkInputHash
   • harnessUrl (permalink into the pinned-commit run.py)

04CI reproduction

Workflow: .github/workflows/benchmark-humaneval-ci.yml — ships in Sprint D W2b (#905).

Triggered by gaia push --benchmark humaneval. On success, the workflow writes an evidence row via gaia dev evidence with:

• --type benchmark-result
• --benchmark-id [email protected]
• --provenance ci-reproduced
• --attestor <workflow-run-url>@<commit-sha>
• --score <pass@1>, --unit pass@1, --run-at <iso8601>
• --dataset-hash <sha256>, --benchmark-input-hash <sha256>
• --harness-url <permalink>

A status check lands on the PR. pending → ci-reproduced promotion is automatic when the workflow passes; on failure, the row remains pending and the merge is blocked (via scripts/validate.py --strict, which is auto-enabled by GITHUB_BASE_REF=main).

05Verifier attestation

A 4★+ Verifier may co-sign a benchmark run directly, bypassing CI when the harness is not automatable (private test set, human graders, hardware lock-in). The evidence row uses:

• --provenance verifier-attested
• --attestor <verifier-github-username>

Verifier-attested rows carry the same datasetHash / benchmarkInputHash requirements and are counted in Trust Magnitude at the same weight as CI-reproduced.

06Trust Magnitude

Per registry/schema/meta.json § evidence.types:

For HumanEval, percentile is derived from the published leaderboard placement at the time of the run.

07First live row

The fixture-based row is the reproducibility bootstrap: the score is deterministic under the stubbed evaluator, the fingerprint hashes are fixed, and any future CI reproduction against the fixture lands the same numbers. Once the full HumanEval dataset lands via workflow_dispatch, the canonical datasetHash populates above and subsequent rows carry the pinned URL as their evidence source.