Jeff Champagne, Field CTO at Cribl – Interview Series

Jeff Champagne, Field CTO at Cribl, brings deep experience across observability, enterprise data, product strategy, and technical architecture. Before joining Cribl, he served as Director of Product Management at Databricks and spent more than seven years at Splunk in senior field product, architecture, and global systems engineering roles. His background also includes systems architecture leadership at BNY ConvergEx, senior engineering work at Premier Technology Solutions, and early engineering leadership at Assist-2-Sell, giving him a broad perspective on how enterprises collect, manage, analyze, and act on operational data at scale.

Cribl is an enterprise telemetry company focused on helping IT, security, SRE, and DevOps teams collect, process, route, store, and analyze observability and security data without being locked into a single vendor. The company positions itself as the “AI Platform for Telemetry,” with products such as Cribl Stream for real-time telemetry collection, reduction, enrichment, and routing; Cribl Edge for vendor-neutral endpoint telemetry collection; Cribl Search for searching data in place; and Cribl Lake for storage. Its broader mission is to give enterprises more control, flexibility, and efficiency as telemetry volumes grow and AI-driven operations require cleaner, more accessible data.

You have worked across Splunk, Databricks, and now Cribl, giving you a rare view into enterprise data, observability, and infrastructure over multiple technology cycles. From that perspective, what feels genuinely different about the observability challenge created by agentic AI?

In application observability, we’ve always dealt with complex, branching execution paths, tracing requests through microservices and correlating logs across distributed systems. In security operations, we’ve tackled high-volume, disparate data sources to detect threats across systems that, while complex, still behave in largely predictable ways. Agentic AI doesn’t reinvent either of those problems; it explodes them simultaneously.

Agents chain decisions, invoke external tools, and spawn sub-agents dynamically in ways that aren’t predetermined at write time. The execution graph isn’t just complex, it can be wildly different every time, and the volume of telemetry required to reconstruct what actually happened grows exponentially with each hop.. Collecting all of the telemetry needed to make agentic systems observable has to be both easy to instrument and cost effective to operate at scale. Otherwise enterprises will make the same tradeoff they always have and reduce visibility to control costs — exactly the wrong time to go blind.

As enterprises embed autonomous agents into workflows, where are the biggest visibility gaps between what an AI agent is asked to do and what actually happens across the underlying systems?

Agents complete workflows that look successful on the surface while quietly making bad tool calls, hallucinating parameters, or drifting from the original goal. The consequences often surface hours later in another system.

The disconnect usually starts at interpretation: slight ambiguity in the original instruction gets resolved differently depending on what the agent retrieves from memory, what tools are available at that moment, or how a prior step in the chain shaped its context. You also have to instrument the downstream systems the agent depends on, because an agent can make a perfectly well-formed database call and still return a wrong answer if the underlying table was never refreshed because an ETL job failed silently upstream.

The agent did exactly what it was told; the data it trusted was just stale.

Why are traditional observability tools often insufficient for monitoring agentic AI systems?

First, they were built around rigid, proprietary schemas and predictable data shapes. Agentic traces don’t conform to those expectations — they are voluminous, highly variable in structure, and include data types like full prompt text, model reasoning chains, and tool call payloads that legacy tools weren’t designed to ingest without significant transformation work. Open standards like OpenTelemetry’s GenAI semantic conventions are emerging to address this, but many legacy platforms weren’t built to take advantage of them.

Second, the scale and cost equation breaks down fast. A single agent handling production workloads can generate more telemetry in an hour than an entire application stack produced in a day, and most traditional platforms charge by volume ingested and stored. That creates an impossible tradeoff — either instrument everything and face runaway costs, or reduce visibility into the systems where you need it most.

When an AI agent fails, how can organizations determine whether the root cause was the model, the retrieval layer, an API dependency, a permissions issue, or a downstream infrastructure problem?

Isolating root cause in an agentic failure is a process of elimination across layers. It only works if your telemetry spans all of them consistently. Latency and error signals on the model spans point to the LLM itself. Retrieval spans tell you whether the context returned was relevant or stale. Tool call spans expose failed API dependencies or permission errors.

As mentioned, you also need telemetry from the downstream systems the agent depends on, because an agent can make a perfectly well-formed call and still produce a wrong answer if the data it trusted was never refreshed. No single signal tells the whole story, which is why correlating across all of those layers in one place is what separates teams that can debug agentic failures quickly from those that are still guessing.

What types of telemetry data become most important when enterprises are trying to understand agent behavior at scale?

The foundation is still MELT — metrics, events, logs, and traces — but the weight shifts considerably when you move into agentic systems. Metrics tell you something is wrong and logs give you the raw record of what happened, but neither one can reconstruct the chain of reasoning an agent followed to arrive at a wrong answer. That’s where traces become the most critical signal.

In a traditional distributed system, a trace shows you how a request moved through services. In an agentic system, it shows you how a model reasoned through a problem, which tools it called, what context it retrieved, where it branched, and what it decided at each step. Without that trace, you’re looking at an output with no way to explain how you got there.

The other MELT signals don’t go away; metrics on token usage and latency matter for cost and performance, and logs from downstream systems remain essential. But traces are what make agent behavior legible at scale.

Many AI discussions still focus heavily on hallucinations. Are enterprises underestimating operational failures such as bad retrievals, broken integrations, latency spikes, or missing context?

Most hallucinations in production aren’t model failures; they are operational failures in disguise. Bad retrieval returns stale or incomplete context, and the model fills the gap with something plausible-sounding. A broken integration means the agent never got the data it needed to reason correctly in the first place. Missing context from a failed ETL job looks like a hallucination but is really an infrastructure problem.

When teams can’t see the difference, they default to what they can control, which is usually the model itself. They iterate on system prompts, add guardrails, and refine evaluation frameworks, and while those techniques have real value, they are treating a symptom rather than the cause. In my experience, the highest-leverage investment is almost always data quality and freshness in the retrieval layer. A well-grounded agent with clean, current data will outperform a heavily prompt-engineered one working from stale or incomplete context every time.

As agents become more autonomous, what should enterprise teams log, trace, or monitor to maintain accountability without overwhelming themselves with noise?

The foundation is still MELT, but accountability for agentic systems requires layering an eval framework on top of traditional telemetry. Metrics, events, and logs surface infrastructure failures and broken integrations that can corrupt agent behavior before the model responds; traces give you the reasoning chain to understand why it arrived at a particular answer. But neither one tells you whether the answer was actually correct, and that’s where LLM-as-judge evaluation fills the gap. The same AI capabilities that power agents can be used to assess them, scoring outputs for accuracy, groundedness, and task completion at a scale no human review process can match.

Pre-production, it acts as a quality gate, catching regressions before they reach users. In production, it becomes a continuous monitor, sampling live traffic and flagging responses that fall outside acceptable thresholds. The key to avoiding noise is recognizing that not every agent workflow carries the same risk, which means moving away from uniform monitoring toward something the industry is starting to call human-on-the-loop governance.  You define behavioral baselines and confidence thresholds upfront, let agents operate them, and reserve human intervention for exceptions.

What role can telemetry pipelines play in helping security, SRE, data, and AI teams work from a shared view of agent behavior?

Security teams, SREs, data engineers, and AI teams are all looking at different slices of the same problem through different tools, and none of them has the full picture on their own. The reason telemetry pipelines have become a standard part of modern enterprise data architecture is that they solve exactly this problem: a single, well-governed pipeline upstream of all your tools ensures every team works from the same normalized, enriched data without bespoke agents and collectors deployed for each team.

That flexibility matters at two levels. The first is data tiering. A well-designed telemetry pipeline lets you match data to the right storage and analysis strategy so you’re not paying analytics-tier prices for data you only need for long-term compliance retention. The second is optionality as the AI tooling landscape continues to evolve rapidly. Enterprises that build their observability strategy around a single vendor’s ecosystem will face expensive migration projects every time a better tool emerges.

What mistakes are you seeing enterprises make when they try to retrofit existing monitoring practices onto agentic AI deployments?

The most common mistake I see is reaching for existing premium observability platforms to collect agent traces without thinking through the cost implications first. Agentic traces are large, variable, and voluminous in ways that traditional application telemetry is not, and most premium platforms price on ingestion and storage. The bill can become a serious problem before teams even realize what’s happening, and the typical response is to reduce what they collect. The result is the worst of both worlds: high cost and incomplete visibility.

The second mistake is applying the wrong mental model to what monitoring is supposed to answer. Traditional infrastructure monitoring asks “is the system up?” and raises alerts when it isn’t. That question is necessary but nowhere near sufficient for agentic systems, where the infrastructure can be perfectly healthy while the agent is quietly making poor decisions, retrieving stale data, or drifting from its original goal. Teams that retrofit existing dashboards and alert thresholds onto agent workflows often convince themselves they have coverage when they’re actually only watching the plumbing.

The third mistake is treating instrumentation as an afterthought. Enterprises that pilot agents quickly and move to production without thinking through observability end up in expensive remediation cycles, rebuilding data collection architecture after the first significant production failure. The teams that scale agent deployments most successfully instrument early, establish baselines while traffic is low and behavior is predictable, and build their evaluation framework before they need it rather than after something goes wrong.

Looking ahead, will infrastructure visibility become a prerequisite for enterprise AI adoption in the same way cloud observability became essential for large-scale cloud migration?

The parallel to the cloud is extremely similar. The most important thing I tell customers when they move infrastructure to the cloud is that they aren’t lifting and shifting. They are deploying a whole new architecture with new capabilities. All of it needs to be instrumented and monitored in ways previous tools were likely not set up for. The enterprises that skipped that step ran into cascading failures they couldn’t diagnose, spiraling costs they couldn’t explain, and outages they couldn’t prevent. Observability became the foundation of trust that made cloud scale possible, and the same pattern is playing out now with agentic AI, moving faster and with higher stakes.

The primary reason agents will need this kind of oversight indefinitely is that they are non-deterministic by design, their behavior emerges at runtime, and their effective capability grows every time you connect a new tool. Static pre-deployment certification cannot cover a system whose action surface changes with every integration. Agents will operate within defined boundaries of trust, and those boundaries will expand over time as observability, policy, and evaluation infrastructure matures. The boundaries themselves will never disappear, and that is not a failure state. It is how every consequential autonomous system in the world already works.

Thank you for the great interview, readers who wish to learn more should visit Cribl.