Before Repairing a Formal Policy, Check What the Words Became

An AI assistant tells a part-time employee with eight months of service that they qualify for parental leave. The formal policy checker behind the assistant agrees. In AWS's worked example, the checker extracted both facts correctly: eight months of continuous service and part-time status. The error sits in the rule that follows. The only rule that applies grants eligibility after six months of continuous service, and no rule excludes part-time employees.

Now change the failure. A user mentions two years of service, and the checker can map that phrase to either of two overlapping tenure variables. The translation models disagree about which one to use, so the checker reports the ambiguity instead of a verdict. Tightening the eligibility rule does nothing for this case, because the dispute is about what the input means, not about what follows from it.

Both examples come from AWS's post of August 3, 2026, which announced automatic policy refinement for Automated Reasoning checks in Amazon Bedrock Guardrails. This article reads that post retrospectively, alongside AWS's current documentation, which may describe details added or reworded since August. The examples demonstrate a software workflow on a sample HR policy. They are not statements about real employment entitlements. What they expose is a maintenance requirement for any team that puts formal checks around AI answers: inspect the translation before choosing the repair.

The proof starts after an interpretation

An Automated Reasoning policy defines variables and formal rules. To check a natural-language response, the service first translates the relevant text into variable assignments, guided by the descriptions attached to each variable. A reasoning engine then checks those assignments against the rules and returns a finding such as VALID, INVALID, or TRANSLATION_AMBIGUOUS.

The two stages answer different questions. Translation asks what the words mean in the policy's vocabulary. Validation asks what follows from that representation. A rigorous answer to the second question does not repair a wrong answer to the first. AWS's testing documentation gives the simplest case: if "2 years" becomes a tenure value of 2 instead of 24, the rules receive a number they can process, but not the number the user stated. Changing a threshold until that example passes would bury the translation error inside a new rule.

AWS's current testing documentation ties its two kinds of test to those two stages. Generated scenarios are variable assignments drawn from the rules themselves, so they test rule semantics with no natural language involved. Question-and-answer tests send realistic input and output text through the whole pipeline, so they also exercise wording and unit conversion. The documentation recommends scenarios first, then question-and-answer tests. Running both lets a maintainer tell a rule that makes the wrong decision from a translation that asks the rules the wrong question.

A rule repair and a vocabulary repair change different things

In the part-time example, the translation is clean and the rule is permissive. This is the case AWS's Iterative Refinement mode addresses. It takes the current policy definition, one or more source documents containing the authoritative text, and optional natural-language feedback. Internally it generates a candidate change, simulates the effect on the policy's saved tests, and adjusts until it converges. The maintainer sees only the converged result: a proposed policy definition and a difference showing each changed rule, each changed variable, and the before-and-after outcome of every saved test. This mode needs at least one attached test to work from.

That is the right place to repair eligibility logic, because the source document states what the policy should say and the failing test shows where the formalization departs from it. The goal is to realign the rules with the document, not to invent whichever restriction makes one example fail. AWS's post also says that a single round may not resolve the failure and may regress other tests. Its advice is to discard the proposal, tighten the feedback and narrow the document to the relevant section, and retry. If the test still fails after two or three rounds, it recommends editing the specific rule by hand, using the finding's translation and rule trace as the guide.

The overlapping-tenure example takes a different path. Ambiguous Variable Refinement analyzes the policy's variables directly, so it needs neither a source document nor attached tests. It proposes clearer variable descriptions and, where two variables describe the same concept, a merge that deletes one and rewrites the rules that referenced it. In AWS's example, the proposal deletes the second tenure variable and expands the surviving description to require complete months, to convert years into months, and to cover phrases such as length of service, time at the company, or seniority.

Here a variable description is part of the system's behavior, not a comment. The translation models use it to decide which variable a phrase refers to and what value to assign. Renaming or merging variables therefore deserves a review of meaning, not only a check that the formal expressions still point at a variable that exists. The expanded description in AWS's example adds a qualifier the earlier wording lacked: the months must be continuous employment. Whether that qualifier matches the HR policy's own text is not something the walkthrough shows, and this article does not judge it. It is the kind of detail a reviewer should check against the source before accepting.

A merge removes a distinction. That is right when the two fields were duplicates. In a hypothetical policy where one variable meant total lifetime service and another meant continuous service since the latest rehire, a merge would collapse two different meanings into one. That case is not in AWS's example. It illustrates why a simpler schema still has to be read against the source document.

A green test suite can still describe the wrong policy

AWS's walkthrough reports that the part-time policy goes from three passing tests out of four to four out of four after refinement. The accepted change deletes the tenure-only eligibility rule and adds a rule that makes part-time employees ineligible. This result shows that the four saved tests now meet their expected outcomes. It does not describe how untested cases behave. The walkthrough shows only the two changed rules and the test outcomes, not the rest of the policy, and this article draws no conclusion about how that policy treats full-time employees.

A clearly hypothetical case shows why the limit matters. Imagine a policy whose intended rule grants a benefit to full-time employees after six months and excludes part-time employees. A repair that denies everyone fixes the false approval while destroying the valid ones. A test set containing only cases that should be rejected would pass in full, and the review screen would show no regression, because nothing in the set asks whether approvals still happen.

The useful test set covers both sides of the decision: cases that should qualify and cases that should not, with the boundary values the policy cares about. It also includes alternative phrasings that should translate the same way. The two halves protect different properties. Eligibility cases exercise the rules, and equivalent wordings exercise translation consistency. AWS's documentation gives the first half of this advice in its own terms, recommending tests for responses that correctly state a rule and tests for responses that misstate it.

AWS's post is explicit that refinement optimizes for passing tests and that this can pull a rule away from what the source document says. It recommends generating a Fidelity Report before and after a change and comparing the two. The report provides a coverage score, an accuracy score, and per-rule grounding that links each rule to the source statements supporting it. The grounding is the most useful part for review, because it turns a vague worry into a specific question: which sentence in the document authorizes the new exclusion, and did deleting the tenure rule remove an entitlement the document still grants?

The post's own best practices call the test results panel the decision-maker for accepting a proposal and the fidelity comparison the check against drift. They check different things: whether the saved cases still get their expected results, and whether the rules still match the source document. The test panel can only answer the questions the test set asks, and the report's scores are a measurement, not a proof that every intended condition survived. Neither replaces reading the relevant passage of the source.

Completion returns a proposal, not a deployed rule

The separation between suggestion and change runs through the programmatic workflow as well. Starting a refinement returns a workflow identifier, not a result. The caller polls until the workflow reaches a terminal state, then retrieves the proposed policy definition. That definition is a complete new draft. To see what changed, the caller compares it with the definition exported before the run started. To commit it, the caller explicitly updates the policy. Until then, nothing has changed. The console wraps the same sequence behind its review screen and its accept and discard buttons.

Accepting a proposal writes it to the draft policy, not to production. AWS's post then calls for re-running the tests, comparing fidelity reports, and creating a numbered version for deployment. The separation matters because generating a candidate and deciding that it expresses the policy's intent are different acts. The post describes the engine as having suggestion authority and the maintainer as having commit authority.

One engineering implication follows from the export-and-compare design. The integration should keep the exported baseline alongside the proposal, because the proposal is a full definition and the change is visible only relative to the definition the engine started from. A difference computed against any other version would describe a different change. The post does not show what happens if the draft is edited while a workflow runs, and this article makes no claim about it.

The product boundary deserves the same care. AWS's current documentation states that a VALID result guarantees validity only for the parts of the input captured by the policy's variables. Statements outside that scope are not validated. Its example is a claim about submitting homework late because of a fake doctor's note, which can come back VALID if the policy has no variable for whether the note is fake. Test results also list the premises and claims that were left untranslated, which shows what a VALID finding did not cover. The checks also run in detect mode only: they return findings and feedback, and the application decides whether to serve the response, rewrite it, or ask the user for clarification. A successful check is therefore neither a truth label for the whole response nor evidence that the application acted on the finding.

Repair the meaning, then preserve the intended decision

Automatic refinement reduces the labor of drafting candidate rules and descriptions. Its value depends on aiming it at the right fault. AWS's post states the same first best practice this article arrives at: open the failing finding and read the variable assignments before picking a mode.

When the variables and values match the input but the conclusion is wrong, the fault is in the rules, and the repair is checked against the source document. When the assignments are wrong, or the translation models disagree about them, the fault usually lies in the descriptions, units, or overlapping concepts, and the repair is checked against what the words were meant to mean. In either case, test the intended approvals as well as the intended rejections, and compare the changed policy with the authoritative text before creating a version.

The part-time employee makes the order concrete. The same wrong answer can come from a misunderstood fact or from a bad rule, and the final finding alone cannot say which. Following the words into their variable assignments can. That step is what keeps an automated repair from converting one visible failure into a quieter change in what the policy enforces.