Observability Summit North America 2026

Reddit processes over 25 billion tracing events per hour across thousands of services. In this talk, we share how we scaled our OpenTelemetry-based distributed tracing platform by 67% in one year—and what broke along the way.

We'll cover our architecture: OpenTelemetry instrumentation across Python, Go, and JavaScript baseplate libraries feeding into Kafka pipelines and ClickHouse storage. You'll learn how we handled an incident that spiked ingestion to well over 10 million spans per second, the sampling strategies we developed to balance cost with debuggability, and why instrumenting three language runtimes simultaneously is harder than it sounds.

Key takeaways:
- Practical patterns for multi-language OTel instrumentation at scale
- Remote sampling strategies that adapt to traffic patterns
- ClickHouse schema design for sub-second trace queries
- Building adoption through cross-functional partnerships, not mandates

Whether you're starting your tracing journey or scaling an existing platform, this talk provides battle-tested lessons from running distributed tracing infrastructure serving one of the world's largest online communities.

OpenTelemetry’s GenAI semantic conventions are evolving quickly. Version 1.37 marked a major shift in how LLM behavior is expressed using standard spans and attributes. While later releases refined and clarified the spec, real-world adoption remains uneven, and “GenAI-compatible” can mean very different things across the ecosystem.

In this talk, I’ll share hands-on lessons from implementing and validating GenAI support in real emitters, including close collaboration with Strands. Implementing the 1.37 spec on both sides surfaced semantic ambiguities that only became clear in practice and ultimately led to stronger implementations.

I’ll also outline the current GenAI instrumentation landscape: Strands emitting 1.37+ compliant spans; OpenLLMetry, which mixes newer conventions with legacy and custom attributes; and OpenInference, which claims OpenTelemetry compatibility but does not emit GenAI semantic convention attributes.

Finally, I’ll show how these gaps surface in practice—teams believing they emit 1.37-compliant telemetry but sending pre-1.37 or non-spec data—and briefly touch on transition guidance like OTEL_SEMCONV_STABILITY_OPT_IN.

Bloomberg runs a managed, multi-cluster Kubernetes platform built atop Karmada to support AI and streaming analytics workloads. This comes with challenges around observability at scale. To meet disaster recovery requirements, we use a multi-region architecture where each Karmada control plane is hosted on management clusters spanning multiple regions. This helps ensure high availability, but also adds complexity related to observability. For example, how do we aggregate and visualize metrics across multiple Prometheus servers when each management cluster has a dedicated Prometheus setup?

This talk covers our multi-region architecture to meet DR requirements and our Prometheus stack with Thanos for global metrics aggregation. We’ll explore how we choose the right signals and define meaningful alerts in a complex multi-cluster environment to curb alert fatigue, while ensuring timely issue detection. We’ll also discuss the challenges of defining SLIs and SLOs in a multi-tenant platform.

Quantiles like p90 and p99 sit at the heart of observability. They define dashboards, drive SLOs, and shape how teams reason about system performance. They are also some of the most expensive metrics to compute, and the cost grows fast as data volumes increase.
To keep up, observability systems rely heavily on approximate quantile algorithms such as sketches and probabilistic data structures, including t-digest. These approaches work well at small and medium scale, but at tens or hundreds of petabytes, things start to creak and limitations become apparent.
We share hard won lessons from operating ClickHouse at extreme scale, where quantile estimation must remain accurate and affordable over hundreds of petabytes of data. We break down the most common quantile algorithms used in observability today, explain their real trade offs, and show when each approach makes sense. We also explore a critical design decision: when quantiles should be computed on the fly at query time versus pre aggregated during ingestion.
The goal is to give you a practical framework for choosing quantile algorithms that scale, rather than blindly relying on defaults that stop working as your data grows.

As large language models (LLMs) move into production, raw metrics alone aren’t enough. This talk presents an open-source AI observability solution built on Open Data Hub (ODH) that deploys LLMs using vLLM and KServe, scrapes inference metrics using Prometheus, and feeds them into a summarization model to generate actionable insights. We’ll demonstrate a working UI that translates low-level metrics like latency, GPU usage, and token throughput into human-readable summaries—giving platform teams an intelligent way to monitor LLMs at scale. No dashboards to interpret—just straight answers from your models about your models.

Instrumentation has never been easier, but are we truly gaining clarity? As data volumes rise, dashboards multiply, and observability costs increase, developers may feel less insight and more friction. Are we collecting telemetry with purpose or just because we can? What problem is this data meant to solve?
This panel brings together practitioners across open standards, developer experience and real-world reliability engineering. The discussion will examine how zero code instrumentation affects workflows and system understanding, how meaningful telemetry improves day to day engineering work and why unfiltered or unstructured data often has the opposite effect. The conversation will cover practical lessons for filtering, dropping, reducing and shaping telemetry so teams maintain visibility without unnecessary volume or cost. Finally, we explore scaling observability across fleets of collectors with an OpAMP server, ensuring consistent signal delivery and manageability as telemetry grows.
At the center is a guiding question: What is the purpose of the telemetry we collect and how do we ensure it remains aligned with developer needs, operational requirements, and system reliability?

Modern observability stacks excel at capturing signals about infrastructure health, application performance, and request flows. Yet one critical class of decisions remains largely invisible: authorization.
In distributed systems, authorization decisions increasingly determine not only whether an action succeeds, but if data is accessed, tools are invoked, or automated agents are allowed to act. These decisions are often evaluated outside application code using Policy as Code frameworks, yet their outcomes are rarely observable in a structured, privacy preserving way.
In this session, we explore how Policy as Code, Open Policy Agent, and the OpenTelemetry project can be combined to treat authorization decisions as observable events. We examine what it means to observe a decision without logging sensitive inputs, how decision structure differs from traditional metrics and traces, and why decision level observability is becoming essential in cloud native and AI driven systems.
Attendees will leave with a conceptual framework for thinking about authorization as telemetry, and a clearer understanding of where observability is heading as systems become more autonomous and policy driven.

During this session we’ll discuss ElastiFlow’s Edge Observability strategy, which includes an OTel native edge processing node with local DuckDB storage for all OTel signals and an agentic AI system that is model agnostic (we often run it with OpenAI’s gpt-oss-20b), exposing its tools through an MCP server.

Instead of just forwarding OTel signals from various Edge collectors, the signals are analyzed and routing decisions are made. Alerts are sent to the Observability Platform right away, while logs are stored locally and analyzed for patterns. Instead of forwarding all logs, we might only care about a few conditions of interest, often correlated with other signals, and send these to the Observability Platform, while less interesting logs can be aggressively aggregated.

With this approach, we were able to reduce the storage and ingest cost of Observability Platforms by half while actually decreasing the mean time to insight.

As infrastructure scales across regions and clusters, Prometheus deployments fragment into isolated islands of metrics—disconnected from logs, traces, and the dashboards operators actually live in.

This talk is for the Prometheus community. If you've wrestled with federation sprawl, alert duplication, or the gap between your metrics and the rest of your observability story, this session is for you.

We'll demonstrate how OpenSearch's distributed data source support lets multiple Prometheus clusters coexist natively alongside logs and traces in a single unified interface, no data migration, no parallel stacks.

You'll learn:

• Unified querying across Prometheus clusters
• SLO tracking wired directly into dashboards
• Application management that finally connects the signals your teams have been operating in isolation

This is about completing the observability loop the Prometheus community has always needed, open, composable, and community-driven.

Saturation has long been the stepchild of "the Four Golden Signals of SRE". While latency, traffic, and errors are directly measurable through metrics like P99, RPS, and 5xx rates, monitoring just how "full" a service is relies on indirect symptoms such as CPU usage or queue depth. Yet, saturation should ideally rather be the first signal to alert, as once it's reached, other signals - latency and errors - spike fast.

The inability to directly observe and mitigate saturation drives excessive safety margins, chronically low CPU utilization and massive compute waste in latency-sensitive and customer-facing systems. This session introduces an open-source approach extending Envoy proxy and its seamless integration through eBPF and Cilium, to provide direct observability into service saturation, by comparing each instance's live number of concurrent requests to its true concurrency limit. We then explore how such direct visualization of saturation can help reduce MTTR and minimize waste.

Agentic AI systems represent a fundamental shift in software architecture: autonomous agents reason, plan, invoke tools, and orchestrate complex workflows without deterministic control flow. This breaks many assumptions behind traditional observability.

When agents independently make decisions, failures no longer follow a single request path. How do you debug emergent behavior across multiple agent steps? How do you analyze and control token-driven costs? How do you ensure reliability when outputs are non-deterministic?

This session explores why observability is a first-class requirement in the agentic AI era and how OpenSearch can act as the analytical backbone for understanding autonomous AI systems in production. We will cover practical techniques for instrumenting agent workflows with OpenTelemetry and indexing traces, logs, metrics, and AI decision artifacts into OpenSearch for deep correlation and analysis.

Attendees will learn battle-tested patterns for tracing agent reasoning and tool usage, investigating failures and hallucinations, monitoring latency and cost signals, and building dashboards that make agentic AI systems transparent, debuggable, and production-ready.

Synthetic monitoring and end-to-end testing often require dangerous levels of access to production systems. Last summer, I nearly emailed my bank details to a team I was training on new testing tools. If I hadn't caught that mistake, I probably would have dumped them into an OTel collector too.

This session explores the security implications of common testing practices, and presents practical alternatives that maintain observability without compromising security.

Attendees will learn authentication and authorization patterns to improve test security across the software development lifecycle.

Implementing mitigations like health check endpoints, synthetic data, and privilege separation spans the full stack of infrastructure, development, monitoring, and governance. Attendees will leave with a shared vocabulary they can use to align business, development, security, and observability teams on safer test traffic in production.