AI Agent Observability: Can You Trust Your Agent's Success Rate?

Most conversations about AI agent observability start from the assumption that businesses running AI agents have no idea what those agents are doing. The evidence points somewhere more uncomfortable and more useful. Businesses do have numbers. The problem is that the standard numbers used to judge an AI agent are less trustworthy than they look, and in several documented cases the measurement layer itself, not the agent, turned out to be the broken component. An agent can score well and still be failing.

That reframes the question a business owner needs to ask. You are not being asked to become an engineering organization or to instrument anything yourself. You are being asked one buying question: when your agent reports success, do you know that number means anything?

AI agent observability starts with the measurement layer itself

The clearest illustration on the public record comes from Anthropic's engineering team, which documented one of its own models scoring 42% on a benchmark called CORE-Bench, then jumping to 95% after bugs in the evaluation were fixed and the model was given a less constrained scaffold. The model itself never changed. The entire 53-point gap came from the apparatus doing the measuring. It is worth noting that Anthropic sells frontier models and is not a neutral party here, but this particular disclosure runs against its own commercial interest: the company is documenting that its own scoring was wrong. Testimony against interest carries more weight than vendor self-promotion.

Academic work published this year reaches the same conclusion from the outside. A February 2026 paper, Towards a Science of AI Agent Reliability, argues that "compressing agent behavior into a single success metric obscures critical operational flaws." Evaluating 15 models across two benchmarks, its authors found that recent capability gains produced only small improvements in actual reliability. A related 2025 paper on building rigorous agentic benchmarks, whose author list includes Percy Liang, Matei Zaharia and Ion Stoica, found that construction flaws in widely used benchmarks lead to under- or over-estimation of agent performance by as much as 100% in relative terms. These two papers share an author, so they should be read as one closely related line of research rather than as two independent confirmations.

The independent confirmation comes from a third group. A June 2026 study titled Reliability without Validity examined 21 models used as automated judges across roughly 541,000 individual judgments. It found that the standard way teams validate these judges, exact-match agreement, does not correct for chance and systematically overstates how well the judge actually discriminates good output from bad.

A dashboard can be perfectly consistent and still measure nothing

The most important finding for a business audience is in that June 2026 study's title. Reliability is not validity. Two of the automated judges the researchers examined were already deployed in production systems and showed test-retest reliability above 0.95, meaning they returned almost exactly the same answer every time they were asked. Those same judges carried severe position bias. They were consistent and they were wrong.

For anyone evaluating a reporting dashboard, that is the whole lesson in one sentence: a number that looks stable week after week can still be measuring nothing. Stability feels like accuracy. It is not the same thing. The study also found that judge rankings shifted by as many as 14 positions depending on which benchmark was used, which means a vendor's score is not portable. A high mark on one evaluation does not predict a high mark on another, a point reinforced by a May 2026 taxonomy of AI agent safety benchmarks that found benchmarks assessing nominally similar concerns often disagree with one another.

The honest counter-argument: most executives say their agents are working

There is a serious objection to everything above, and it deserves a direct answer rather than a quiet omission. PwC's AI Agent Survey, fielded in May 2025 among 300 senior executives, found that 79% said AI agents were already being adopted in their companies, and that of those adopting, two-thirds (66%) said the agents were delivering measurable value through increased productivity. That is not a picture of organizations flying blind, and it comes from a survey of the exact population this article is describing.

The reconciliation is straightforward, and it is the crux of the argument. Self-reported executive confidence is not the same thing as verified measurement. The research above is specifically about the distance between a confident number and a valid one. Nothing in the PwC survey establishes that the 66% figure rests on measurement that would survive the scrutiny those papers apply. Equally, nothing establishes that it would not. Both statements can be true at once: agents are delivering real value in many organizations, and the instruments used to confirm that value have known, documented failure modes.

One more piece of intellectual honesty belongs here, because the skepticism cuts in both directions. In researching this article we looked specifically for a study demonstrating that teams who instrument and monitor their agents achieve better business outcomes than teams who do not. We could not locate one. That is not proof that instrumentation fails to pay off, and it should not be read that way. It does mean that the commercial case for buying observability tooling currently rests on reasoning rather than on demonstrated business results, which is worth knowing before a purchase order gets signed. PwC's own writing on the subject is carefully hedged in the same direction, noting that without observability organizations "may operate with limited visibility" into production AI agent systems.

Four questions to ask before you trust an agent's score

The February 2026 reliability paper proposes twelve metrics organized across four dimensions: consistency, robustness, predictability and safety. Those dimensions translate cleanly into four questions any business owner can ask a vendor or an internal team without writing a line of code.

• Does it do the same thing twice? Run the same request through the agent on different days. Consistency is the cheapest thing to check and the most commonly assumed.
• Does it hold up when the inputs get strange? Real customers send malformed, incomplete and unusual requests. Performance on clean test cases predicts very little about performance on messy ones.
• When it fails, would we notice? This is the dimension most often skipped. An agent that fails loudly is a manageable problem. An agent that fails quietly, returning confident output that happens to be wrong, is the expensive one.
• How bad can a single failure get? Bounded error severity is a business question, not a technical one. It determines how much autonomy an agent should have over a given process.

These four questions are where AI agent observability earns its keep for a smaller organization. They require no platform purchase and no engineering project. They require deciding, in advance, what a working agent would look like in your specific business.

What to do if you have not deployed an agent yet

It is worth being clear about who is actually in this situation. According to the U.S. Census Bureau's Business Trends and Outlook Survey, between December 2025 and May 2026 fewer than 20% of firms with four or fewer employees reported using AI at all, and AI use showed no significant change among firms with fewer than 20 employees while rising among larger ones. Among firms with at least 250 employees, 37% reported using AI. For most small businesses, the agent has not been deployed yet. The measurement decision is still ahead of them, which is the best possible position to be in.

If that describes you, the practical guidance from practitioners is consistent and it argues against spending money first. Independent industry analysis makes the case that dashboards and data collection are the table-stakes part of evaluation, while the genuinely hard work is deciding what to evaluate and why. A parallel analysis of the evaluation-tooling market reaches the same conclusion about where the difficulty actually sits. For resource-constrained teams, practitioners Hamel Husain and Shreya Shankar recommend designating a single domain expert as the arbiter of quality rather than buying tooling, and warn that the most common mistake is starting with ready-made, fashionable metrics such as hallucination or toxicity scores. That advice is prescriptive rather than measured, but it costs nothing to follow.

Two further signals are worth tracking. Among organizations that do build agents, LangChain's State of Agent Engineering survey of 1,340 respondents reports human review used by 59.8% of organizations against automated AI judging at 53.3%. That survey samples teams already sophisticated enough to use an agent framework, so its adoption figures should not be read as representative of businesses generally, but the finding that human review still leads runs against the commercial interest of a company selling automated evaluation, which is what makes it worth repeating. Separately, a vendor-neutral instrumentation standard is taking shape: OpenTelemetry's semantic conventions for GenAI agent spans define a common vocabulary for agent activity. That document is still marked "Status: Development" with no published release, so the correct posture is to ask a prospective vendor whether they support it, and to treat the area as unsettled rather than solved.

The takeaways

• An AI agent can post a strong success score while failing in production. Multiple independent research groups documented this in 2026.
• A measurement that is highly consistent is not necessarily correct. Two production-deployed automated judges showed test-retest reliability above 0.95 alongside severe bias.
• Benchmark scores are not portable between benchmarks, so vendor scores from different evaluations cannot be compared directly.
• Executives broadly report that their agents deliver value. That confidence is self-reported and is not the same as verified measurement.
• Ask four questions instead of buying a platform: does it do the same thing twice, does it hold up on strange inputs, would we notice a failure, and how bad can one failure get?
• Deciding what to measure is the hard and valuable part. Collecting the data is comparatively easy.

One question runs through all of it, and it belongs in every conversation about automation, including the ones you have with us: who checked the checker? A vendor who can answer that clearly is telling you something real about how their systems are built. Before you commission an AI agent from anyone, agree in advance on what working looks like and how you will confirm it.

Talk to our team

Have more questions or want to get in touch? We help small and mid-sized businesses across Charlotte, Raleigh, Asheville and Philadelphia design AI agents and automated workflows that come with an honest answer to how you will know they are working. Contact Idea Forge Studios to start the conversation. We look forward to hearing from you.

Prefer to speak with a member of our team? Give us a call at (980) 322-4500. We are available during our business hours to assist you. Have a detailed question or prefer to write to us? Send us an email at [email protected] and our team will get back to you as soon as possible.

If you are earlier in the journey, our guides to agentic AI orchestration, AI agentic workflows for dynamic automation and strategic AI agent transformation cover the decisions that come before this one.

Citations

1. Anthropic – "Demystifying evals for AI agents" (January 9, 2026)
2. arXiv – Rabanser, Kapoor, Kirgis, Liu, Utpala & Narayanan – "Towards a Science of AI Agent Reliability" (February 18, 2026; revised June 2, 2026)
3. arXiv – Zhu et al. – "Establishing Best Practices for Building Rigorous Agentic Benchmarks" (July 3, 2025)
4. arXiv – Norman, Rivera & Hughes – "Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias" (June 17, 2026)
5. arXiv – Li, Fung, Li, Ismail & Iqbal – "Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents" (May 19, 2026)
6. PwC – "PwC's AI Agent Survey" (survey fielded May 2025)
7. PwC – "Observability is the key ingredient in making AI and agents work for you" (February 19, 2026)
8. U.S. Census Bureau – "Large Firms With at Least 20 Employees Biggest AI Users" (May 26, 2026)
9. The Pragmatic Engineer – "A pragmatic guide to LLM evals for devs" (December 2, 2025)
10. Enterprise AI Trends – "Evals Startups Are Not Enterprise Ready" (June 8, 2025)
11. Lenny's Newsletter – Hamel Husain & Shreya Shankar – "Building eval systems that improve your AI product" (September 9, 2025)
12. LangChain – "State of Agent Engineering" (June 12, 2026)
13. OpenTelemetry / CNCF – "Semantic Conventions for GenAI Agent Spans" (living document; verified July 21, 2026)