Methodology · Frozen post-merge

How a score becomes evidence.

Every benchmark row that touches a Gaia badge carries a public commit a provenance lane. Self-attestation is rejected at the schema. Reported public claims can count after the human gate; rejected rows score zero.

01Provenance

Every benchmark-result evidence row carries a provenance lane. The simplified #1419 model has three lanes, with legacy aliases accepted during migration.

self-attested is intentionally absent and forever rejected — both at the schema and in scripts/validate.py. A candidate source becomes reported only after the human gate approves it, or rejected if the source is disputed/blacklisted.

pending is a legacy no-scoring alias and remains a hard error under --strict, which the validator auto-enables when the base ref is main. New human-approved public claims should use reported; blacklisted or disputed rows should use rejected.

Machines classify. Humans approve or reject.

02Reproducibility fingerprint

Two SHA-256 hashes are required for verified rows:

• datasetHash — SHA-256 of the raw dataset. A revision produces a new hash and therefore a new row.
• benchmarkInputHash — SHA-256 of (dataset + prompt template + harness config). Same dataset, different config, different row.

Both are 64-character lowercase hex. Uppercase, short, or non-hex values are rejected at both the CLI and schema layers for verified rows. Reported rows do not need runAt, datasetHash, or benchmarkInputHash; their attestor/source is the public claim accepted by the human gate.

03Trust Magnitude contribution

The benchmark-result evidence type is one of the highest-weight signals in the registry:

• weight: 1.4
• gradeCeiling: S
• allowedLayers: [generic, named]
• inheritMultiplier: 0.15
• Magnitude = normalized base score × lane multiplier. Base score uses percentile when present; otherwise score is normalized by unit.
• Freshness: 50%/year decay (half-life ≈ 1 year).

Trust Magnitude computes the final benchmark score internally; do not persist a finalScore. Verified rows use a 2.0× lane multiplier, reported rows use 1.0×, and rejected/pending/unknown/candidate/retired rows use 0×. The existing benchmark cap applies after the lane multiplier.

Schema-valid does not automatically mean scoring-valid. Phase 2B of /ev-pipeline checks each benchmark-result row against registry/benchmark-sources.json: rejected or unknown catalog sources are zero-score; verified rows need reproducibility fields; reported rows need human approval and a public attestor/source.

04Adding a new benchmark

1. Author the harness. Deterministic Python, pinned deps, emits results JSON with score, unit, runAt, datasetHash, and benchmarkInputHash. Lands at scripts/benchmarks/<id>/run.py.
2. Author the CI workflow. .github/workflows/benchmark-<id>-ci.yml runs run.py on the submitted commit and, on success, writes an evidence row via gaia dev evidence with --provenance verified and --attestor <workflow-url>@<sha>.
3. Register the benchmark id. Add a landing document at docs/benchmarks/<id>/ (see humaneval-v1).
4. Wire the leaderboard. Point at docs/api/v1/benchmarks/<id>.json.

The benchmarkId field is semver-ish: <name>@<version> or <name>/<subset>@<version>. Examples: [email protected], humaneval/[email protected], [email protected], mmlu-5shot@2024-03.

A rename is a major schema bump. Names in this document are load-bearing.

05Frozen invariants