How do we know a skill is real?

Reproducibility fingerprints. Provenance labels. A public commit SHA for every score that touches a badge. Self-attestation is rejected at the schema; mirrored numbers cite but never count.

Every row carries datasetHash, benchmarkInputHash, and an attestor identity. Provenance is one of ci-reproduced, verifier-attested, mirrored, or pending — only the first two contribute to Trust Magnitude.

External benchmarks

Public tasksets, run inside a Gaia CI harness or mirrored as citations from a named upstream leaderboard. Every score links back to the commit that produced it.

How rows are ingested, hashed, and gated at merge time is documented in the benchmark methodology.

In design

Gaia Skill Bench

A frozen taskset that scores the skill—its harness, its prompts, its refusal behaviour—independent of the model executing it.

  1. 40%
    Performance Task success rate against the frozen taskset hash.
  2. 30%
    Reliability Consistency across repeated seeded runs.
  3. 20%
    Triggering Fires when it should; stays silent when it shouldn't.
  4. 10%
    Efficiency Tokens, calls, and latency to reach the answer.

Submissions enter through the community harness. Gaia CI re-executes the run against the frozen taskset hash and, on success, writes a row with provenance: ci-reproduced. Everything else stays in pending until a Verifier or a green workflow promotes it.

Read the vision doc Follow issue #960

Original proposal by @rico-favor in issue #960.