AI agents now read documents, call tools, and take action inside your systems. When one does the wrong thing, the first question is simple: what happened? Without AI observability, nobody can answer it.

The tools to answer it already exist. Most AI-native tracing and evaluation tools, including Arize Phoenix, Langfuse, and Traceloop’s OpenLLMetry, are built on or export OpenTelemetry. That means AI spans can flow through the same collectors and pipelines as the rest of your telemetry.

The market just confirmed it. On October 1, 2026, Dynatrace closed its acquisition of Arize. Days later, New Relic announced native AI Evaluation, building on the OpenTelemetry-based AI monitoring it made generally available in August. AI-native tracing and eval are being absorbed into mainstream APM platforms rather than staying a standalone tool category. The question for security teams is whether that meets their needs.

What AI observability captures

Traditional monitoring tells you a service is slow. AI observability records each step an agent takes as linked spans: the prompt, the content it retrieved, the decision it made, the tool it called with which arguments, and what came back. Evaluation tools then score those outputs for accuracy and quality.

For an agentic system, that trace is the closest thing you have to an audit log.

Why it is critical

• Investigation. Reconstructing a prompt-injection attack means seeing the poisoned content, the reasoning step, and the resulting tool call as one chain.
• Disclosure timing. Incident reporting clocks depend on knowing when anomalous behavior started. You need the spans to still exist weeks later.
• Accountability. Tying an agent action back to the person or service that invoked it is the core question in any review.
• Early warning. New Relic’s 2026 Observability Forecast found that in 42% of organizations, disruptions are still discovered through manual checks or customer complaints. An agent quietly doing the wrong thing is even harder to spot.

Does it meet security requirements? Not by default.

AI observability is being bought on operations budgets and configured for latency charts and invoice size. Those defaults leave clear gaps:

• Evaluation is not detection. Evals measure quality against expected behavior. A manipulated agent can produce a well-written, well-grounded answer while its tool call exfiltrates data. The score looks fine.
• Sampling drops the evidence. A 1% trace sample gives good latency percentiles and almost certainly discards the one session you need to investigate.
• Retention is too short. Ops keeps 7 to 30 days to control cost. Investigations and disclosure obligations can run for months.
• Traces are now sensitive data. Prompts, retrieved documents, and tool arguments contain PII, proprietary code, and sometimes credentials, stored in a platform that is broadly readable across engineering by design.
• Identity is missing. Without agent identity and invoking principal on every span, a trace cannot answer the question security actually asks: who did this?
• Alerts go to the wrong team. Anomalies land in ops channels that the SOC never reads.

What cyber teams need to own

1. Get a seat on the observability purchase. Requirements added after signature become change orders.
2. Define the AI span schema. Require agent identity, invoking principal, tool name and scope, model version, and retrieval source IDs.
3. Set a security retention class. Keep security-relevant AI spans unsampled and on your IR and legal retention period, costed separately from ops telemetry.
4. Control the data at the collector. Redact sensitive fields in the OpenTelemetry collector before data leaves your environment, scope access to trace payloads, and log who queries them. Because the AI tools already speak OpenTelemetry, the collector is the one enforcement point that survives a vendor change.
5. Add adversarial evals. Put prompt injection, tool-scope escalation, and exfiltration cases into the eval suite and make them blocking in CI.
6. Route signals to the SOC. Anomalous traces belong in the detection pipeline, not only on a dashboard.

Observability is necessary, not sufficient

AI observability closes the visibility gap that has made AI incidents nearly impossible to investigate. It does not, on its own, close the security gap. That part still belongs to the cyber team.

Here is the test: if you cannot pull the complete tool-call chain for a single agent session from last month, you do not have AI observability. You have AI dashboards.

Di1 helps enterprise and federal teams design AI observability that holds up to an investigation, from span schema to retention and access controls. Book a consultation to review your AI telemetry before the defaults are locked in.