How do we know a skill is real?

Reproducibility fingerprints for verified rows. Human-gated reported public claims. A simple lane multiplier for Trust Magnitude. Self-attestation is rejected at the schema; rejected benchmarks score zero.

Every row carries a benchmark lane: verified (CI/verifier reproduced, 2.0×), reported (public claim or mirrored evidence approved by the human gate, 1.0×), or rejected (blacklist/no scoring, 0×). Legacy aliases are still accepted during migration.

External benchmarks

Public tasksets, either verified inside a Gaia CI/verifier harness or reported from a public source after the human gate. Every score links back to the source that supports it.

How rows are ingested, hashed, and gated at merge time is documented in the benchmark methodology.

In design

Gaia Skill Bench

A frozen taskset that scores the skill—its harness, its prompts, its refusal behaviour—independent of the model executing it.

  1. 40%
    Performance Task success rate against the frozen taskset hash.
  2. 30%
    Reliability Consistency across repeated seeded runs.
  3. 20%
    Triggering Fires when it should; stays silent when it shouldn't.
  4. 10%
    Efficiency Tokens, calls, and latency to reach the answer.

Submissions enter through the community harness. Gaia CI re-executes the run against the frozen taskset hash and, on success, writes a row with provenance: verified. Public claims can enter as reported only after the human gate; disputed rows become rejected.

Read the vision doc Follow issue #960

Original proposal by @rico-favor in issue #960.