Agent Observability Maturity

Agent observability maturity is the progression from manual vibe checks and isolated traces toward production feedback loops that connect human annotation, automated scoring, trace analysis, external system state, eval datasets, and quality improvements ^[src-088].

Key points

• Phil Hetzel describes evals and observability as one systems problem: before launch, teams use evals to become confident; after launch, they use observability to remain confident ^[src-088].
• The first stage can be human review, but the valuable artifact is the justification behind thumbs-up or thumbs-down labels because that extracts domain knowledge for later automated graders ^[src-088].
• Mature teams identify real failure modes, convert them into LLM-as-judge or deterministic scoring functions, and pull production or UAT traces back into offline eval runs ^[src-088].
• Tool-using agents add complexity because evaluation may need the full trace, including tool calls, MCP calls, token and cost behavior, external system state, and whether CRUD actions were safely simulated or mocked ^[src-088].
• Agent observability differs from traditional observability because the important question is often not "did the service respond?" but "did this stochastic multi-step system pursue the right task, with the right evidence, at acceptable cost and risk?" ^[src-088].

Related entities

• Braintrust

Related concepts

• Continuous Agent Evaluation
• LLM Observability
• Spec-Driven Agent Testing
• Agent Forensics
• Harness Engineering

Source references

• ^[src-088] AI Engineer late-May 2026 channel update (48 transcripts, 2026-05-15 to 2026-05-31)