Three Numbers Hide Inside One AI Support Resolution Rate

An AI support tool evaluated by Magic Hour reported 75 percent automation. When the company counted reopened issues, silent abandonment and human handoffs as failures, its figure was closer to 40 percent, according to Runbo Li, co-founder and CEO of Magic Hour.

Li did not name the tool or supply a sample size. His account describes one company's evaluation, not an independent audit of a vendor. The difference does raise a practical question for a support manager: when a dashboard says an issue was resolved, which event did it count, and what happened to the customer afterward?

The tempting response is to decide which figure is right. The more useful one is to notice that the two sides counted different events. The vendor's figure came from its own definition of a resolved conversation. Magic Hour's figure followed the customer's problem afterward, treating a reopened issue, a silent exit or a transfer to a person as a failure. A third figure, the one on the invoice, describes which events a vendor charges for.

So there are three different questions to answer. How often did the AI complete the work assigned to it? Did the customer eventually get the problem solved, including any help from a person? Which event qualified for a charge? None of the three answers is necessarily dishonest. Each answers a different question, which is why a single percentage cannot settle an argument between them. A customer can receive useful automated help, still need a human and generate a billable event along the way. Treating those observations as one measure makes it hard to decide whether to change the AI, the handoff or the commercial arrangement.

Five practitioners' written responses to PSEEDR describe different observation windows and escalation rules. Read alongside the public documentation of Intercom's Fin, they show why the definitions behind a rate need to stay visible. Fin appears here as a documented example of one vendor's rules. Nothing in this article identifies the tool Li evaluated.

Start with the work the bot was allowed to attempt

Consider a hypothetical business receiving 100 support conversations. Its AI attempts 50 and resolves 40 under a stated rule. Dividing 40 by the 50 attempts gives an 80 percent resolution rate on attempted work. Dividing the same 40 by all 100 incoming conversations gives 40 percent. The resolved count did not change; the denominator, the group used as the base of the calculation, did. One rate describes success on assigned work, while the other describes how much incoming work the AI completed.

Routing decisions can change both figures. If more difficult cases go straight to people, the AI's success rate on its remaining attempts can rise even while it resolves fewer incoming conversations overall. If it takes on more difficult work, it may resolve more conversations while succeeding on a smaller share of its attempts. The team needs the attempted and resolved counts alongside the percentage to distinguish a change in workload from a change in performance.

Intercom's documentation of a change to its Fin performance metrics shows this arithmetic in a vendor's own words. The page defines three measures. Involvement Rate is the share of all conversations in which Fin was involved. Resolution Rate is the share of involved conversations that Fin resolved. Automation Rate is resolutions divided by all conversations, and the page states that Automation Rate equals Involvement Rate multiplied by Resolution Rate. The vendor describes the first as a measure of opportunity, the second as a measure of performance and the third as a measure of business value.

The change removed from the involved group those conversations in which Fin was active but never had a chance to answer, which the page calls Fin Constrained. Its worked example has 1,000 conversations, 250 resolved by Fin, Fin active in 750 and Fin given an opportunity to answer in 500. Involvement Rate falls from 75 percent to 50 percent. Resolution Rate rises from 33 percent to 50 percent. Automation Rate stays at 25 percent, because the number of resolutions did not move, and the page says billing is not affected.

Using the page's rounded figures, Resolution Rate rises by 17 percentage points, from 33 percent to 50 percent, without a single additional resolution. The documentation presents the new rate as a more accurate measure of performance when Fin had a chance to answer. A buyer comparing periods still needs the same definition on both sides: otherwise a reporting change can look like a service improvement. A percentage can move because the model improved, because the work changed or because the denominator changed, and each explanation points to a different next step.

The list of reasons a conversation can be constrained is also instructive. The page names escalation guidance that sends a topic to a teammate, a workflow that triggers before Fin responds, usage limits, a customer who never asks a question, and a customer who requests a handoff to a teammate before asking anything. Under the new definition those conversations leave the Resolution Rate denominator.

Victor Smushkevich, founder of Tested Media, a marketing agency for service businesses, uses a broader base for his own measurement: every conversation the bot opened, including the ones where the person asked for an agent in the first message. "Pull those out and the rate looks great for the wrong reason," he writes.

Smushkevich's measure asks how much of the work reaching the bot it completes. Fin's Resolution Rate asks how often it succeeds when it has an opportunity to answer. Those are different questions. Smushkevich's base would match Fin's all-conversation Automation Rate only if the bot opened every incoming conversation. If some conversations bypass the bot, his base would be smaller than the business's full inbound volume.

Li starts with every inbound support conversation initiated by a user. Pratik Singh Raguwanshi, manager of digital experience at CXrove, uses all unique initiated support sessions. These broad measures help show the AI's contribution to the incoming workload; a rate restricted to attempted work can help assess how it performed on that assignment. The practical question is which number appeared in the sales conversation, which one appears at the renewal review, and whether anyone in the room knows that they differ.

The customer's problem can outlive the chat

Ending a chat is observable. Ending the problem that caused it is not, at least not from the transcript alone.

A customer who goes quiet after an answer may have succeeded. They may have postponed trying it, switched to email, phoned instead or given up. Each respondent has a rule for that silence, and the rules diverge. Raguwanshi counts silent abandonment, where a user stops responding without reaching a solution, as a failure. Li does the same and is blunt about the alternative: "That's not resolution. That's silent abandonment dressed up as success." Dora Bloom, chief revenue officer at iotum, writes that counting a ticket closed for lack of reply as a win "is endowing silence with a meaning it never had."

Smushkevich chooses a third path. Someone who stops replying after a bot answer has not told you anything, so he labels the conversation unknown and tracks that category next to the resolved number. That is a different claim from calling silence a failure. It says the data does not support either conclusion, and it makes the size of the uncertainty visible. A large unknown category tells a manager that the business has weak visibility into what happens after an answer, which is a finding in itself.

Fin's pricing rules take a fourth position. Under Fin's pricing documentation, a customer who disengages for 24 hours after Fin's last answer produces an assumed resolution, and that is a billable outcome. The same page draws a sharp boundary around the rule. If Fin's last message was a clarifying question rather than an answer and the customer never replies, no resolution is recorded and nothing is billed. Silence after an answer counts; silence after a question does not. A buyer who knows that boundary can check which kind of silence dominates their closed conversations.

Each of these rules is a choice with a cost. Treating every silence as failure undercounts self-service that worked. Treating every silence as success claims knowledge the transcript does not contain. Keeping an unknown category avoids either assumption, but it leaves a manager with a group that is hard to act on until someone looks at the cases, and a transcript alone may still not settle them. There is no neutral default. The useful move is to state which rule is in force and why it suits the product's customers.

Then there is the question of where the customer goes next. Smushkevich matches a second contact on the same topic through email or a form back to the original conversation, by hand or by contact ID. That shifts the object being tracked from a thread to a person's problem. Fin's documentation says that if a customer returns to the same conversation seeking more help, even across billing periods, the resolution is deducted. That is a real protection, and the documented adjustment is confined to the same conversation. The documentation does not describe matching a new email thread about the same unfinished request back to the original one. Smushkevich's advice follows directly. Vendor dashboards report what the vendor's tool did; your own count should come from the ticket or inbox data you control.

Matching needs judgment as well as a contact ID. The same customer can come back with an unrelated question, and reclassifying every later contact as a failure would penalize the bot for continued use of the product. Counting every new thread as a fresh opportunity hides repeated attempts at one problem. The useful record links the customer, the issue and the sequence of contacts, and it keeps confirmed, unconfirmed and reopened outcomes apart.

Let the window close before you read the number

The respondents use different observation windows. Raguwanshi describes a confirmed solution followed by 24 to 48 hours without re-engagement through any channel. Li requires a substantive answer and no reopening of the same issue within 72 hours. Bloom recommends seven days. Smushkevich also sets seven days, adding: "Anything shorter flatters the bot."

The disagreement is informative because a window defines which later events the measure can detect. For example, a customer can test an instruction for changing a setting at once. Checking advice about a charge on next month's statement may take longer. Li explains his 72-hour choice with an observation from his own users: most people who received a bad answer either retry within a day or churn silently. A short window returns a result sooner and misses the slow failures. A long window catches more returns and needs careful matching so that new problems are not counted as old failures.

Whatever window a team chooses, it has to let that window close. Yesterday's conversations cannot yet meet a seven-day rule. If last month's group had a full week to reopen and this week's group has had only a day, the newer group can look better before anything has improved. The remedy is to hold the original group of issues fixed and update its outcomes as the window closes. When a customer returns with the same problem inside the window, the first thread flips from resolved to unresolved; the return is not a new thread to be counted as a second success. Li describes exactly that rule at Magic Hour: the original ticket flips back.

Bloom turns the same idea into a buyer's request. Ask for the same number two ways, raw and net of reopens and human requests, along with the window used to produce it. If a vendor can only show the raw figure, she writes, that is a marketing figure, not an operations figure. The raw and net pair is useful because the gap between them is the quantity a single percentage hides, provided both figures share the same starting group and the adjustment rules are stated. Otherwise the gap could mix a change in the denominator with a change in which outcomes qualify.

Li adds another internal check: if the customer has not confirmed resolution and does not return to the product within seven days, Magic Hour flags the conversation as unresolved. That is a deliberately conservative rule, not proof that absence means failure, and its meaning depends on how often customers normally use the product. A business whose customers appear once a quarter would read the same seven days of absence differently, and might keep Smushkevich's unknown category instead. The rule is not the point. Stating the assumption and matching it to how the product is actually used is.

A handoff can be good service and a bad statistic

There is a simple way to raise a containment number, the share of conversations that never reach a person: make reaching a person harder. Whether that improves support is a separate question, and two of the respondents changed it in opposite directions.

Li says Magic Hour's rule routes a thread to a human once a user has sent more than three messages without the AI resolving the issue. The team tried raising that threshold to five. Customer satisfaction dropped noticeably, and they reverted to three within a week. His reading: "Users don't get more patient with AI the longer the conversation goes. They get less patient."

Raguwanshi moved the other way, from a three-strike rule, in which the bot attempted three rephrasings before offering a human, to a single-strike handoff after one failure to identify intent or on detecting frustration keywords. The three-strike design, he writes, produced "angry transfers" in which agents began the interaction at an emotional deficit. He reports improved long-term resolution after the change. Neither account supplies an effect size that would let another company predict its own result. Both describe the same operational choice: whether one more bot turn moves the task forward or delays the next useful step.

Rick Elmore, founder and CEO of Simply Noted, states the rule in its simplest form: "if a customer asks for a person, they get a person, immediately, with no second attempt by the bot." He describes his company as small and speaks to the principle rather than a dataset. Li applies the same principle to categories rather than requests. Billing disputes, account access problems and anything where the user is already frustrated go to a person immediately, with no AI gatekeeper, and he reports faster resolution and measurably lower churn for those cases. He does not give the measurement, so the claim stands as his report. It is also the clearest case among the five replies of more human involvement producing better service.

Bloom argues that a request for a person is the customer telling you the automation did not land, and that scoring it as a successful handoff gives the vendor credit the customer never gave. On the AI's own score she is right: a conversation that ends in a request for a person is not an AI-only resolution, and no definition should count it as one. That request does not settle the customer's eventual outcome, though. An early transfer can be the best thing that happened in the conversation, which is the result Li and Raguwanshi describe. The two records only conflict when a team tries to collapse them into one number.

Smushkevich adds the test that connects them. In the follow-up automations he configures, a direct request for a person routes to a human right away with the transcript attached. He says he cannot offer a measured outcome number and will not make one up. What he watches instead is how often the human thread starts with the customer re-explaining the problem. That is a different question from whether the handoff event fired. It asks whether the information the automation collected was usable by the next person.

Consider a hypothetical request. A bot identifies the product, establishes the symptom and collects an order reference before reaching its limit. If the agent who takes over can see and use that context, the automation contributed even though it did not finish. If the customer has to repeat every step, the automated portion may have added effort rather than removed it. Both cases show up on a dashboard as one handoff. They are not worth the same, and the difference is visible in the transcript.

So a team can keep separate records without contradiction. AI-only completion is one. Resolution after human assistance is another. The re-explanation rate on transfers is a quality measure for the seam between them. A lower automation percentage can accompany faster, less frustrating service when transfers happen at the right moment and carry their context; the percentage alone cannot show that, which is why the other records matter.

The invoice uses its own definition

The invoice is a third accounting system, and it defines the event the supplier charges for. That event need not match either the model's score or the buyer's view of a solved problem.

Fin's public documentation gives a specific example of billing definitions. Its outcome definitions describe two service outcomes. A resolution is counted when, after Fin's last answer, the customer either confirms the answer helped or leaves without asking for more. A procedure handoff is counted when Fin successfully executes a procedure that the workspace has configured to end in a handoff to a human or a workflow. Both are billable outcomes, charged at most once per conversation.

The documentation then distinguishes that configured handoff from ordinary escalation. When Fin hands off because of its default behavior, for instance because the customer asks for a human or shows frustration, or because of workspace-level escalation rules, that is an escalation and is not billed. The same page states that Fin's Resolution Rate is the percentage of conversations Fin resolves without human involvement. A procedure handoff is therefore an event that appears on the invoice but not in the resolution-rate numerator, by the vendor's own design.

The FAQ on the Intercom-hosted page adds configuration-specific cases that cut the other way: human involvement with a charge. If a teammate joins a conversation after Fin has already answered and the customer has not asked for more help, the conversation is considered resolved and billed. If the customer asks for help but a collect-information setting is on and the customer then goes inactive without supplying that information, the documentation says the conversation is still marked resolved and charged. These are narrow rules for particular settings, and this article is not a billing guide. The point they make is general. Human involvement and a billed resolution are not mutually exclusive, and a billed event is not the same as an autonomous one.

For the buyer, the question is whether the billed step did useful work. A procedure that verifies an account, collects the details a person will need and then hands the case over can be worth paying for even though it is not an AI-only resolution. Conversely, a large count of billed outcomes does not establish a corresponding fall in human workload, because some of those outcomes end with a person still working. Keeping billing events, autonomous completions and final customer outcomes in separate columns lets a team value each one rather than argue about a shared label.

What to put on the table at the next review

The practical procedure is to start with a fixed group of incoming issues and follow it through the whole process. For that group, record how many the AI attempted, how many it completed under the chosen rule, how many went to a person and how many remain unconfirmed. Revisit the group after its window has closed and update the outcomes. Present the resolution figure raw and net, as Bloom asks, and state the window beside it. Track the re-explanation rate on transfers, as Smushkevich does.

Then compare billed events with AI-only completions and final customer outcomes, including outcomes after human help. A charge for a completed procedure can be consistent with valuable automation and further human work. Seeing these events separately helps a team assess what the purchased step contributed, instead of treating every billed event as a problem solved.

Read the changes rather than celebrate a percentage. Did the bot solve more work, or did routing give it easier work? Did repeat contacts fall, or are the newest cases too young to have reopened? Did handoffs get more useful, or did customers stop asking? Did billed outcomes rise because the system completed more valuable procedures, or because the vendor's outcome definition and the team's dashboard metric were never the same thing?

The accounts do not establish a universal observation window or message threshold; the respondents disagree on both, and each works from a different product and customer base. What they share is the insistence that definitions stay explicit long enough for someone to tell whether a change helped. That means a comparison the team can explain: the same starting population, the same success rule and enough time for the chosen outcome to be observed. Where those conditions change, identify the change before attributing a higher rate to better AI.

Raguwanshi states the service goal plainly: "Success should be measured by the reduction in total customer effort, not merely the reduction in cost for the enterprise." The strongest support result is not that a customer disappeared from a chat. It is that the reason they asked for help stopped requiring work from them and from the business. AI-only completion, the customer's outcome and the invoice each see one part of that. The dashboard should be built so that a team can see all three, including the cases where a person finished the job.

Reporting note: The five accounts were supplied in written responses to PSEEDR and were not independently reproduced. Smushkevich said he does not sell a support bot and had no vendor tie to disclose. Li did not name the tool behind his reported comparison. Vendor definitions are drawn from the linked documentation consulted for the original reporting; they do not establish the outcome of any individual customer's issue. Hypothetical arithmetic and proposed checks are PSEEDR's analysis.