mmlu@2024-03 · Citation-only
The MMLU citation.
Fifty-seven academic subjects, five-shot. Every score in this section is copied from a public leaderboard — rendered so the record is complete, permanently excluded so it never inflates a badge.
mirrored. Scores here do not count toward
Trust Magnitude. See section 03 for the ladder.
01What is MMLU
Massive Multitask Language Understanding measures a language model's knowledge and reasoning across 57 academic subjects — mathematics, history, law, medicine, computer science — in a 5-shot setting. The model sees five example questions with answers before being tested on new ones.
Original paper: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020).
02Snapshot source
Scores in this registry are copied from a static snapshot of the
HuggingFace Open LLM Leaderboard
dated 2024-03-01. The snapshot file lives at
scripts/benchmarks/mmlu/snapshot.json.
| Field | Value |
|---|---|
benchmarkId | mmlu@2024-03 |
unit | pct (0..100 percentage accuracy) |
sourceSnapshotDate | 2024-03-01 |
runAt | 2024-03-01T00:00:00Z |
03Provenance ladder
The Gaia registry requires reproducible provenance for Trust Magnitude contribution. MMLU scores in this snapshot are cited from a public leaderboard — they were not produced by a CI-executed harness in this repository and have not been co-signed by a 4★+ Verifier running the model directly.
| Provenance | TM | How achieved |
|---|---|---|
ci-reproduced | Counted | CI workflow re-ran the harness on the same commit |
verifier-attested | Counted | A 4★+ Verifier co-signed the run |
mirrored | Excluded | Cited from a public leaderboard |
pending | Excluded | Awaiting CI reproduction |
A cited number can be stale. It can be measured under different prompts, different splits, a different tokenizer. Counting it would inflate the badge.
The leaderboard renders mirrored rows with a "Cited" badge to surface the distinction without hiding the data. This is deliberate: the record is complete, the score is legible, and TM stays clean.
04Refreshing the snapshot
- Visit the leaderboard.
- Export current 5-shot MMLU averages for the skills in
snapshot.json. - Edit
scripts/benchmarks/mmlu/snapshot.json; bumpsourceSnapshotDate. - Run
python scripts/benchmarks/mmlu/ingest.py --dry-runto preview. - Run
GAIA_OPERATOR_OVERRIDE=1 python scripts/benchmarks/mmlu/ingest.pyto write. - Regenerate
docs/api/v1/benchmarks/mmlu.json. - Open a PR on a
review/meta/branch.
05API projection
Machine-readable row data is served at
/api/v1/benchmarks/mmlu.json.
The index of all registered benchmarks lives at
/api/v1/benchmarks/index.json.