mmlu@2024-03 · Citation-only
The MMLU citation.
Fifty-seven academic subjects, five-shot. Every score in this section is copied from a public leaderboard — rendered so the record is complete, marked reported so it counts at the 1.0× lane only after the human gate.
reported. Scores here count at the reported
1.0× Trust Magnitude lane after human gate approval. They are not Gaia
CI/verifier reproduced.
01What is MMLU
Massive Multitask Language Understanding measures a language model's knowledge and reasoning across 57 academic subjects — mathematics, history, law, medicine, computer science — in a 5-shot setting. The model sees five example questions with answers before being tested on new ones.
Original paper: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020).
02Snapshot source
Scores in this registry are copied from a static snapshot of the
HuggingFace Open LLM Leaderboard
dated 2024-03-01. The snapshot file lives at
scripts/benchmarks/mmlu/snapshot.json.
| Field | Value |
|---|---|
benchmarkId | mmlu@2024-03 |
unit | pct (0..100 percentage accuracy) |
sourceSnapshotDate | 2024-03-01 |
runAt | 2024-03-01T00:00:00Z |
03Provenance ladder
The registry requires reproducible provenance for Trust Magnitude contribution. MMLU scores in this snapshot are cited from a public leaderboard — they were not produced by a CI-executed harness in this repository and have not been co-signed by a 4★+ Verifier running the model directly.
| Provenance | TM | How achieved |
|---|---|---|
verified | 2.0× | CI-reproduced or verifier-attested; legacy aliases ci-reproduced/verifier-attested |
reported | 1.0× | Public claim or mirrored leaderboard accepted by the human gate; legacy alias mirrored |
rejected | 0× | Blacklisted, disputed, pending, candidate, retired, or unknown |
A cited number can be stale. It can be measured under different prompts, different splits, a different tokenizer. Counting it would inflate the badge.
The leaderboard renders reported rows with a "Reported" badge to surface the distinction without hiding the data. This is deliberate: the record is complete, the score is legible, and TM stays clean.
04Refreshing the snapshot
- Visit the leaderboard.
- Export current 5-shot MMLU averages for the skills in
snapshot.json. - Edit
scripts/benchmarks/mmlu/snapshot.json; bumpsourceSnapshotDate. - Run
python scripts/benchmarks/mmlu/ingest.py --dry-runto preview. - Run
GAIA_OPERATOR_OVERRIDE=1 python scripts/benchmarks/mmlu/ingest.pyto write. - Regenerate
docs/api/v1/benchmarks/mmlu.json. - Open a PR on a
review/meta/branch.
05API projection
Machine-readable row data is served at
/api/v1/benchmarks/mmlu.json.
The index of all registered benchmarks lives at
/api/v1/benchmarks/index.json.