Why Passing AI Code Review Doesn’t Mean Your App Works

Two pull requests pass review. One changes invitation permissions; the other changes workspace switching. Together, they can still leave a user looking at controls for the wrong role. Each diff may look correct while the combined application behaves incorrectly.

That is the testing gap this article examines: the distance between reviewing a change and verifying the behavior of the release that contains it. Closing it requires evidence about the integrated build, its deployment configuration, and the outcomes users can actually reach.

The gap matters as AI code review improves. At GitHub Universe on October 28, 2026, GitHub introduced Copilot code review Max and reported the highest F1 score for high-severity issues on ReviewBench at the time of the announcement. That is a useful development for teams trying to review more code without overlooking important defects. It is also a specific claim about a specific evaluation, rather than a measure of whether your next release works. GitHub Universe announcement.

For a small team, the response should be manageable: use AI review where it helps, then connect the important findings and checks to the exact build you intend to ship. This article develops that approach through a hypothetical invitation workflow. It does not report a real incident or a test run performed by HiTest.

What an AI review score can tell you

GitHub introduced ReviewBench on October 5. Its corpus contains 219 pull requests from 187 public repositories across 19 languages. Its reference findings combine multiple sources, including human review, static analysis, and models, evaluated under a shared rubric.

Precision describes how many reported findings are valid. Recall describes how many known valid findings the reviewer identifies. F1 balances the two. Those measures help distinguish a noisy reviewer from one that finds useful problems consistently.

ReviewBench separates grounded metrics, evaluated against existing labels, from augmented metrics that also assess unmatched findings. GitHub uses grounded recall for its headline cross-system comparison because augmented recall changes its denominator according to what each system discovers. That methodological choice matters when interpreting rankings. ReviewBench methodology.

For your team, a benchmark is a starting point for evaluation. Ask whether the kinds of changes, languages, and failure consequences resemble your work. A reviewer that catches a subtle control-flow defect may still lack the product requirement that distinguishes an owner from an administrator in your application.

The operational question is what happens after a finding arrives. Can someone reproduce it? Does the proposed fix address the cause? Does the fix introduce a different failure? Measure that loop on your own changes before turning a leaderboard position into a release policy.

GitHub's responsible-use documentation also identifies missed issues, false positives, and inaccurate suggestions as limitations, and recommends reviewing and testing changes. A clean review should therefore be recorded as the outcome of that review, without silently expanding it into a guarantee about the application. Copilot Agents application card.

A green PR can become stale evidence

Consider a fictional SaaS application with two pending changes:

• PR A moves invitation permissions into a shared membership service.
• PR B changes workspace switching to reuse cached membership information.

Both changes can look reasonable in isolation. The permission tests in A might construct a fresh membership for each request. The interface tests in B might use an owner account throughout. After both changes land, a session that switches from an owner workspace to a viewer workspace might temporarily display controls using stale role information.

Whether that becomes a security defect depends on the server-side authorization. If the server rejects the operation, there may still be a confusing interface defect. If the server trusts the stale context too, the consequence is more serious. A screenshot of a visible invitation button cannot settle which case applies.

This is a reason to preserve the identity of the tested candidate. The result “invitation checks passed” loses meaning if nobody can establish which combination of changes it covered.

Open full size

GitHub's merge queue addresses part of this problem by checking a proposed change with the latest target branch and queued predecessors. Required GitHub Actions checks need the merge_group event when using that workflow. Availability depends on the repository and plan. The important principle is testing the integrated candidate, whether a team implements that through a queue or a simpler controlled release process. Managing a merge queue.

A successful combined-code check still leaves deployment questions. Did the intended build reach the preview? Are the relevant flags enabled? Did the migration run? Does sign-in return to the same environment? Our preview deployment testing guide covers those setup details.

Give each verification layer a clear job

An efficient release process assigns questions to the layer best able to answer them. Otherwise teams pay for long browser runs that cannot observe the failure, or expect a code reviewer to infer runtime behavior that depends on missing context.

Code and architecture review

Use review to examine authorization boundaries, ownership of state, failure handling, and the scope of a change. In the invitation example, useful questions include:

• Where is the current workspace identity obtained and validated?
• Which operation checks the caller's current role?
• Can a cached decision survive a role change?
• What happens if invitation creation succeeds but delivery fails?

Ask for a concrete failure path, affected code, and relevant assumption. A plausible warning becomes more useful when it tells an engineer how to investigate it.

Deterministic service and integration checks

Exercise permissions and state transitions directly. Test that a viewer cannot create an invitation even when the interface is bypassed. Check that a revoked token cannot create membership. If retry behavior matters, verify the chosen duplicate-handling policy against controlled records.

Keep expected outcomes grounded in product decisions. Having an agent generate tests from its own implementation can reproduce the same misunderstanding in both places. A second agent can challenge assumptions, but changing models alone does not establish independence. Supply the requirement and ask which observations could disprove the implementation.

Running-application checks

Use a browser to connect the interface, session, navigation, and visible result. Switch workspaces, revisit the page, refresh, and exercise failure recovery with the intended roles.

Playwright recommends testing user-visible behavior, isolating tests, controlling test data, and using resilient locators and assertions. Those practices reduce avoidable test noise; they do not remove the need to choose the right expected behavior. Playwright best practices.

When an assertion fails, preserve enough context to investigate it. A recording shows the visible sequence. A trace can add action details, DOM snapshots, and network information. Neither automatically proves a database invariant or an email's arrival in an inbox. Playwright Trace Viewer.

Work through one release decision

For the fictional application, assume the product owner has agreed on these rules: owners can invite, viewers cannot, revoked invitations cannot grant access, role changes take effect on subsequent protected requests, and delivery status must reflect what the system actually knows.

The following is a proposed test plan, not a report of completed checks.

Each row has a boundary. A pending invitation in a database establishes creation, not delivery. An email provider accepting a request establishes another step, not that the recipient read the message. Keep the interface language and the test report consistent with the observation available.

The second-tab case needs an explicit policy too. Must an open page update immediately, on focus, or only after the next request? Whatever the interface does, define the server's authorization behavior separately. Otherwise one tester may call an outdated button a failure while another calls it acceptable, and both may overlook the more consequential request.

Suppose the combined candidate correctly rejects a viewer's request but shows a success toast because the client mishandles the rejection. The useful report names that mismatch, includes the account role and workspace transition, and separates the rejected operation from the misleading message. “Invite test failed” leaves too much work for the engineer receiving it.

For a broader case set, see user invitation test cases and API/UI consistency testing.

Define when evidence expires

A passing result belongs to a combination of code, configuration, data, and execution context. Time is only a rough proxy for freshness. A result from yesterday may remain relevant to an unchanged path; a result from five minutes ago may already be invalidated by a permission fix.

Use a change-based rule:

1. Identify what changed since the recorded run.
2. Map the change to affected behaviors and assumptions.
3. Rerun checks whose evidence depends on those assumptions.
4. Record the remaining uncertainty and who accepts it.

For the invitation example, a new membership cache requires another cross-workspace check. A delivery configuration change requires revisiting delivery evidence even if the commit is identical. An unrelated copy edit usually does not justify rerunning every authorization scenario.

This approach needs judgment, especially around shared code. When the affected area is unclear, broaden the checks rather than documenting certainty you do not have. The retesting guide for SaaS changes provides a practical starting point.

The pressure is visible at much larger scales. In October, Anthropic reported increasing from about 2,500 daily merged changes in May to more than 30,000 in September, and described investment in post-merge and pre-deployment testing. Its source-control system addresses an unusual operating scale and is explicitly not a general replacement for Git. The relevant lesson for a small team is to inspect where verification becomes stale, rather than copy the infrastructure. Anthropic's engineering report.

Keep a compact release-evidence record

A small record attached to the candidate can make the release decision easier to review. Start with fields people can actually maintain:

Candidate: commit SHA + build/deployment identifier
Environment: exact preview URL + relevant configuration
Context: test accounts, roles, workspaces, and fixture identities
Required behavior: links to agreed acceptance conditions
Completed checks: result + evidence link + execution time
Failures: reproducible steps + observed and expected outcome
Not checked: missing coverage or unavailable dependencies
Decision: owner + unresolved risks + release or hold

This is an illustrative format, not a requirement to introduce a new system. A PR description and linked CI artifacts may be enough. Avoid copying credentials, customer records, or sensitive session data into it.

For the fictional release, an unresolved cross-workspace authorization failure should block release under the agreed policy. A provider outage that prevents a delivery test needs a different decision: restore the test dependency, defer the affected change, or explicitly accept the limited evidence through the team's process. Marking an unavailable check as passed conceals the decision instead of making it.

Keep failures reproducible and remaining work explicit. Our browser test failure report guide describes the information that helps a developer act on a result.

Measure the quality of the decision

More comments, tests, or recordings can mean more activity without better release decisions. Useful team measures include:

• Actionable findings: which review findings were reproduced and led to a justified fix?
• Escaped regressions: which user journeys failed after release, with what severity?
• Current evidence: which critical journeys have relevant results for this candidate?
• Diagnostic cost: how often do failures require retries, and how long does explanation take?

Track these as learning signals, not quotas. A reviewer should not be rewarded for producing noise, and a test suite should not look healthier because failures were retried until green. Preserve the initial failure and the reason a retry changed the outcome.

HiTest can contribute browser exploration, generated test cases, and recorded workflows to this process. Review those cases against the intended behavior and connect the results to the candidate being assessed. Keep service-level invariants and external delivery checks with tools that can observe them.

Before the next release, choose one critical journey. Record the candidate, identify the decisions its checks actually support, and name the important behavior that remains unverified. That small exercise often reveals more useful work than adding another undifferentiated green check.