Observability Summit North America 2026

Building AI agents is easy, but knowing why they fail is hard. Traditional APM tools were designed for request-response services, not autonomous agents that reason, plan, and execute multi-step workflows. When your agent makes unexpected decisions, standard metrics and traces don't tell you why.

This session introduces Eval-Driven Development, which focuses on building reliable agents through continuous observability and evaluation. Using OpenSearch AgentHealth, a new open-source platform for agent observability, we'll walk you through the full agent lifecycle of building, observing, improving, and repeating. We'll share a case study comparing two production root-cause-analysis agents. One was built with observability from day one and shipped in a 6 weeks, while the other was retrofitted later and took 12 months to reach production. You'll learn how we used agentic evaluation to score agent outputs and improve accuracy over time.

You'll walk away with patterns for instrumenting agents with OpenTelemetry, techniques for evaluating full decision sequences (not just outputs), and a framework for shortening your development timeline by building observability in from the start.