Practitioner Model Benchmarking Methodology

Nate Herk's framework for evaluating AI models with real agentic tasks, beyond official benchmarks.

Methodology

1. Design 3–5 identical, representative tasks (e.g., build a personal brand site, build a 3D game, simulate an ecosystem)

2. Run each task as a one-shot prompt across all models — no iteration, no follow-up

3. Capture from JSONL logs: wall-clock runtime, input tokens, output tokens, cost, tool call count

4. Assess output quality subjectively against the same rubric

5. Cross-reference with official benchmarks to triangulate ^[src-012]

Key insight: output token efficiency matters most

Output tokens are priced higher than input tokens. A model that produces the same result with 3.5x fewer output tokens (GPT 5.5 vs Opus 4.7) is significantly cheaper in practice even if nominal pricing looks similar. ^[src-012]

Official benchmarks used for cross-reference

• Terminal Bench 2.0 (agentic coding speed/accuracy)
• SWE-bench Verified (real GitHub issue resolution)
• GDPval, Frontier Math, Cyber Gym ^[src-012]

Statistical reporting layer

• Anthropic's eval-statistics paper adds a formal layer that practitioner benchmarks often omit: every benchmark score should be treated as a noisy estimate with standard errors and confidence intervals, not a single deterministic capability fact ^[src-067].
• When two models are run on the same questions, Paired Difference Model Evals should report mean differences, standard errors, confidence intervals, and score correlations rather than only two independent headline scores ^[src-067].
• If benchmark items are grouped by passage, task, repository, or another shared unit, Clustered Standard Errors In Evals are needed to avoid overconfident conclusions ^[src-067].
• Power analysis can decide whether a benchmark is large enough to detect the size of model gap the evaluator actually cares about ^[src-067].

Related entities

• Gpt 55 — April 2026 benchmark subject vs Opus 4.7
• Claude Opus 4 7 — April 2026 benchmark subject
• Gpt Image 2, Imagen 3 — image model benchmark subjects
• Anthropic — source for the statistical eval-reporting recommendations

Related concepts

• Token Economics — cost model underpinning the benchmark analysis
• Statistical Model Evaluations
• Question Universe Eval Framing
• Paired Difference Model Evals
• Clustered Standard Errors In Evals
• Experiment Statistical Power

Source references

• ^[src-012] Nate Herk — Video editing & content creation cluster (2026-04-15 to 2026-04-23)
• ^[src-067] Anthropic – "A statistical approach to model evaluations" (2024-11-19)