Historical source material recovered on 2026-09-09. Capture dates and source limitations apply; recovery is not fresh verification.

AI Evaluation

AI evaluation covers the methods used to measure model or agent performance, reliability, robustness, and alignment with intended outcomes.

Key facts

• Ornith-1.0 claims performance improvements for self-scaffolding coding agents, but AI Watch treats those claims as requiring careful benchmark and reward-hacking scrutiny ^[src-182].
• Doctolib uses shared evaluation and observability as the first scaling primitive for regulated AI products, enabling PMs and healthcare experts to participate directly and supporting hundreds of reported experiments per day ^[src-236].

Related

• Model Benchmarking Methodology
• Reward Hacking
• Agentic Coding

Historical recovered snapshot — 2026-08-03

Preserved from pr-123. Documentary recovery on 2026-09-09; claims below retain their historical source scope and were not freshly verified. Current synthesis above remains authoritative.

Google Cloud's Gen AI evaluation service combines adaptive prompt-specific pass/fail rubrics, static shared rubrics, deterministic metrics, custom functions and task-specific agent measures ^{[src-265][src-266]}. Its third-party evaluation notebook separates model inference from evaluation so API-hosted, Model Garden MaaS and BYOM endpoints can be compared on common application data and criteria ^[src-268].

ROUGE and BLEU measure reference overlap, not general factuality or business value. Strong evaluation stacks combine task-specific rubrics, action/tool metrics, deterministic checks where appropriate, representative production data and human review of material failures.