Building an AI agent in five minutes — yeah, everyone knows that's possible now. The real question is what comes after: when you've got dozens or hundreds of agents running simultaneously in production, who's actually responsible for managing them? Maryam Ashoori, VP at IBM watsonx, says it plainly: "The energy has shifted. The focus now is on operating AI agents at scale with confidence."
What is Agent Ops? The operational layer for monitoring, tracing, and governing AI agents in production
Why now? By late 2025, companies had deployed hundreds of agents with no management infrastructure — and operational risk was exploding
The core shift: From build time to run time — building is solved; operating is the new battleground
What Is It?
Agent Ops is essentially DevOps for AI agents. It's a new discipline focused not on building agents, but on running them safely, at scale, inside real business systems.
Why did this suddenly become necessary? The timeline tells the story.
- 2023: Exploration
Most companies treated generative AI as exploratory investment. Value came from narrow use cases — summarization, classification, code generation. - Early 2024: The Agent Hype Cycle
As LLMs gained the ability to call APIs, "agentic AI" exploded as a concept. CIOs started demanding agents without any clear plan for what to do with them. - Late 2025: Reality Check
Companies found themselves with dozens or hundreds of agents running across different platforms — built by developers, by business teams, by external vendors — all mixed together and becoming unmanageable. - 2026: The Agent Ops Era
The focus shifted from build time to run time. Monitoring, governance, and observability became core capabilities.
IBM's Ashoori puts it bluntly: "If a model hallucinates and calls the wrong tool, and that tool accesses unauthorized data, you've got a data breach." This isn't a matter of a bad answer — it's an operational incident.
And this isn't just IBM's view. According to LangChain's State of Agent Engineering report, 89% of organizations have already adopted agent observability, and 62% have implemented detailed step-level tracing. "How do we manage what we've built?" has become an industry-wide question.
Gartner projects that by 2028, roughly one-third of all interactions with generative AI services will happen through autonomous agents. When agents are everywhere, operating without a management framework isn't an option.
What Changes?
The "build era" and the "run era" demand completely different capabilities.
| Build Time | Run Time | |
|---|---|---|
| Key Question | How fast can we build an agent? | Can we trust this agent in production? |
| Failure Type | Prompt errors, wrong model choice | Causal failures across multi-step reasoning chains |
| Debugging | Check input → output | Full session tracing (trace → span → tool call) |
| Security | API key management | Accountability for agent autonomy, policy enforcement |
| Cost Management | Model API call costs | Per-task cost attribution (which step is burning money?) |
| Success Metrics | "It works!" | Task completion rate, tool selection accuracy, human escalation rate |
Traditional LLM monitoring was simple — just check input and output. Agents are different. A single request gets broken into multiple steps, each involving model calls, tool calls, and data source access. Figuring out where something went wrong means tracing the entire execution path.
Arize AI frames the problem this way: "Agent failures don't happen in a single call — they happen across multi-step causal chains." Step 2 returns a bad search result. Step 4 passes wrong arguments to a tool. Step 5 silently corrupts the state. But the final answer at step 8 looks perfectly reasonable. They call this a "False Success" — and it's the most dangerous failure mode of all.
Heads Up: Right now, only about 19% of organizations in production are focused on observability and monitoring. Agents are multiplying fast — but the control tower is nearly empty.
Getting Started
Adopting Agent Ops starts with a mindset shift: stop treating agents as "software" and start treating them as "operational assets."
- Build your tracing layer first
Before scaling up your user base, instrument session IDs, trace IDs, per-step spans, tool inputs/outputs, and latency/cost. Tools like LangSmith, Arize Phoenix, and Langfuse handle this layer. - Turn failures into evaluation datasets
When something breaks in production, convert it into a regression test case. The goal: the same failure never happens twice. Braintrust and LangSmith both let you add a failed trace to your eval dataset in one click. - Use agent-specific metrics to make deployment decisions
Track more than just answer quality — measure task completion rate, tool selection accuracy, unnecessary tool call rate, recovery rate after tool failure, and human escalation rate. - Decouple governance from your build system
IBM's Ashoori emphasizes that the system you use to build agents and the system you use to govern them should be separate. Regardless of what framework you used or where the agents are running, the same monitoring, evaluation, and optimization standards should apply. - Do a weekly trace review
Production agents quietly degrade when no one's watching. Check for drift in traces and evaluation metrics every week, and keep the loop running — turning failures into test coverage.
Pro Tip: Here's a quick self-assessment for "production ready" — Can you replay a failed agent run step by step? Can you see every tool's inputs and outputs? Do you know the total cost of a single task? Can you detect loops, retries, and dead-end branches?
Deep Dive Resources
Agent Ops Observability Tools Compared A guide comparing Arize AX, LangSmith, Langfuse, Braintrust, AgentOps, and other major agent observability tools in 2026 — broken down by architecture (proxy vs. SDK). arize.com
LangSmith Agent Observability Guide LangChain's practical guide to agent observability — covering tracing, multi-turn evaluation, and AI-assisted debugging. langchain.com
The Production Agent Deployment Loop dev.to
Why AI Agent Implementations Fail in the Real World blog.dfinite.ai
