An Agent's Successful Tool Call Does Not Prove the Record Is Right

The agent checked an overdue order, read the compensation policy and opened a support ticket. Its decision to deny compensation matched the customer's account terms. Then it marked the ticket solved, even though the carrier exception remained open.

The task is a synthetic one from Microsoft's ThinkingBox benchmark, described in a post by Tuhin Kundu that Microsoft and Hugging Face published jointly on October 3 and updated on October 22. The required ticket status was hold. The agent wrote solved. Nine well-formed tool calls and a reassuring final reply did not make that last field correct. The post notes a second fault, that the customer never received a real answer to her question, but the executable check that fails is the single status field.

The example identifies a useful boundary for enterprise evaluations. A tool response can establish that an operation executed. It does not, on its own, establish that the operation produced the result the task required. To test that, the evaluator needs an account of what should have changed and what should have stayed unchanged.

Define the result before watching the route

The updated post describes 507 synthetic business workflows. Each task specifies a starting backend state, a user goal, available MCP tools, a domain policy and executable checks on the ending state. A simulated user holds private context, such as a booking reference, and supplies it when asked. Each attempt gets a fresh, isolated backend, so another attempt's records do not accidentally satisfy its requirements.

At the end of an attempt, the framework extracts the side effects and checks them against the required outcome. The authors describe checks for wrong values, missing required effects and unintended extra effects. The evaluator accepts any sequence of tool calls that reaches an acceptable ending state. It does not prescribe one route.

Grading the ending state makes the ticket example more precise than a complaint that the agent misunderstood the customer. The evaluated failure has a location: the final status field. Denying compensation correctly does not justify closing the case. Compensation eligibility and the ticket's operational status are separate decisions, even when the same conversation touches both.

For a team building its own tests, the implication is to write those decisions down separately. In a similar workflow, one assertion could check the required ticket status, another could check whether a refund was issued, and another could check that unrelated tickets were not changed. These are illustrative conditions, not a claim about the exact assertions behind ThinkingBox's example. They turn a broad request to handle the case into outcomes a reviewer can inspect.

A clean finish can still be a failed task

The benchmark authors quantify the gap in a common-set comparison covering 121,680 valid trials across 12 models. Of 79,853 attempts that failed the executable checks, 67.24 percent nevertheless ended cleanly, invoked a state-changing tool and reported no final tool error.

The denominator matters. That percentage describes failed attempts in this comparison. It is not the fraction of all enterprise requests that fail, and it does not estimate the error rate of an agent deployed by a reader. It shows that a substantial share of these observed failures would survive a coarse check for a clean exit plus a database write.

The trace of tool calls is still useful. It can show the lookup that supplied a record identifier, a failed precondition or the argument passed into an update. Those details help locate the cause. The ending-state check answers a different question: did the required result exist when the attempt finished? Keeping both lets an engineer distinguish an incorrect request from a correct request that failed to take effect.

Checking only that something changed is too weak. A write to the wrong customer's ticket can satisfy a generic "database updated" signal. Checking only the intended field can also be too weak if the same run creates an unauthorized refund elsewhere. The useful test needs the relevant record identities and the side effects that matter to this task. More logged activity is not a substitute for choosing those conditions.

The checker also has a specification

A deterministic test is only as informative as the requirement it encodes. If a business process allows several valid ending states, a test that accepts just one can reject legitimate work. If it inspects only a ticket status, it cannot establish that the explanation to the customer was adequate.

ThinkingBox makes that second limit explicit. Of its 507 tasks, 477 are judged on state alone. Thirty add narrow response rubrics for requirements that have no clean database value, such as whether the agent disclosed that an outcome was not guaranteed. It would therefore be wrong to describe the entire benchmark as checking only database fields or dispensing with semantic judgment.

For an enterprise team, this suggests reviewing the test specification with the people who own the workflow. Which outcomes are acceptable? Which extra changes would be harmful? Which promises exist only in the message sent to the user? The answers determine what the evaluation can establish. These are proposed design questions, not an additional validation study conducted by the ThinkingBox authors.

The golden state, the expected ending state the checker compares against, and the grading internals also stay on the evaluator's side in the published design. The agent receives the task, dialogue and tool schemas. That separation matters. An agent that can inspect or modify its expected answers could satisfy the machinery without showing the intended workflow capability.

Decide when the state is ready to judge

An isolated benchmark can finish an episode and inspect its backend. A live workflow may leave work in a queue or depend on a carrier, payment processor or another service. An accepted request and a completed downstream effect can then occur at different times.

A production evaluation borrowing this approach needs to name the observation point. For a queued update, the useful evidence might include whether the job completed, what record it changed and whether the result was still pending at a stated deadline. Reading too early can label unfinished work a failure. Accepting the queue acknowledgement as success can hide a job that never completes. These are implementation considerations drawn from the mechanism; the post does not demonstrate a general solution for asynchronous production systems.

Observation is also different from prevention. Reading back a refund after it was issued can expose an error, but it does not reverse the payment. Preventing an unacceptable change requires an appropriate control before or during execution. A final-state test is evidence about an outcome, not an automatic rollback mechanism or a replacement for permission checks.

The post proposes changes such as better retry handling, a smaller tool surface and human approval for changes that cannot be cheaply reversed. It states that their improvement has not been measured on this benchmark. That distinction should survive adoption. Finding a failure gives a team a testable target; it does not prove that a proposed repair works.

Start with one outcome the current check misses

A useful first experiment is to take one completed-looking failure and express the missing condition directly. Preserve its initial records, run the agent, then compare the required result and prohibited side effects with what remains. Retain the trace to explain a mismatch, while keeping the ending state as evidence for the specific outcome under test.

The public benchmark supplies synthetic reconstructions, not real customers or an independently reproduced production success rate. Its model rankings and repeated-run results answer additional questions, but they do not resolve a poorly specified definition of success.

The ticket was not wrong because the agent failed to use a tool. It was wrong because a successful write recorded the wrong conclusion. An enterprise test that can identify that difference gives the team something concrete to fix: the condition under which the case may be closed.

Reporting note: This analysis relies on the ThinkingBox blog post first published on October 3, 2026, updated on October 22, and accessed on November 1. It does not assume that every detail visible in the accessed version was present on the original publication date. PSEEDR did not run the benchmark. Proposed test conditions and the discussion of asynchronous observation are PSEEDR's analysis.

Source

The Agent Said It Was Done. The Database Disagreed., Microsoft and Hugging Face.