How to Interpret Z-Score Results From AI Watermark Detectors

A z-score from a KGW-family AI watermark detector measures a statistical pattern. Interpreting it correctly requires attention to the detector’s key, the amount of scoreable text, and the threshold used. For content teams, the central distinction is between evidence of a watermark and proof of provenance. The first can inform a decision, but it cannot substitute for the second.

What KGW watermarks actually measure

KGW watermarks bias token choices as the model generates text, without attaching a marker to each word. At each step, the preceding context token is hashed with a secret key to divide the vocabulary pseudorandomly into green and red lists. Green-list tokens receive an additive logit bonus, δ, while red-list tokens remain unchanged. The model still samples from the resulting distribution.

The partition changes with the preceding context. A token can therefore be green in one position and red in another, so an isolated word reveals little. The detector reconstructs those partitions using the same key and counts the green tokens across scoreable positions. It then runs a one-proportion z-test against a binomial null, asking whether the count exceeds what chance would predict. A large positive result indicates an unusually large excess.

Three parameters define the mechanism and its detection conditions. The green-list fraction, γ, sets the chance baseline, δ controls the generation bias, and T counts eligible token positions. These have different roles in the final score. The z-score subtracts the expected green count from the observed count and divides that excess by the standard deviation under the null.

Why the z-score grows with document length, not word count

The detection statistic is z = (x − γT) / √(Tγ(1−γ)), where x is the observed green count. An observed green count above the expected count gives a high positive z-score when the excess is large relative to the denominator. This arithmetic measures accumulated evidence. It also uses eligible token positions.

For a stable green-token excess rate, the numerator grows with T while the denominator grows with √T. Doubling the eligible text therefore multiplies the expected score by roughly 1.4. Length strengthens a consistent signal, but not linearly. A high score on a long document and a low score on a short passage need not indicate different generation biases.

Text type complicates that comparison because factual writing gives the model less freedom to choose among alternatives. Green-token excess can consequently be sparse even in a long passage. A z-score of 2 on 100 tokens remains ambiguous because the evidence is thin. The same score on 600 tokens is weaker support for a sustained watermark bias, although it does not exclude a watermark. Length and register must be read together.

What detection thresholds actually represent

A threshold turns the statistic into a decision based on a chosen policy for errors. A common deployment threshold is z ≥ 4, at which this test reports a watermark signal for an excess that would be very unlikely under the null assumptions.

Raising the threshold leads to fewer false positives but more missed watermarks, especially in short passages. Lowering it reverses that tradeoff. A media fact-checker and a compliance auditor may assign different costs to those errors, so a threshold appropriate for one setting need not suit another. LIDet at ICLR 2025 found that a fixed threshold achieved an average detection success rate of just 25% in model-infringement detection, while an adaptive method achieved substantially higher accuracy.

Calibration also depends on the test’s assumptions. The treatment of token choices as independent Bernoulli trials simplifies real text, where repeated n-grams, copied passages, and context dependencies can violate independence. A nominal threshold therefore cannot be interpreted separately from how scoreable positions are selected and how dependencies affect the null distribution.

False negatives: the cases where a real watermark produces a low score

A real watermark can fall below threshold when a passage supplies too few eligible positions. The generation bias may have been at work throughout, yet the detector has too little excess built up to tell it apart from chance.

Constrained text creates a different diagnostic problem. Lists, citations, code, and highly formulaic prose limit the available token choices, pulling the green-token rate toward the null despite the applied bias. Light human editing can also weaken detection while preserving much of the original text. Changes to wording, punctuation, or phrasing may leave some signal in place but cut the eligible evidence enough for the score to fall below threshold. A negative result does not identify which mechanism was responsible.

Outputs from models that predate watermarking deployment present another limit, although these are not missed instances of a real watermark. They contain no mark to detect. Their low scores cannot distinguish them from human text on that basis. The defensible conclusion is narrower than “human-written,” namely that the detector did not find sufficient evidence under this key, threshold, and text length.

False positives: when the score flags content that isn't fully AI-generated

A detected watermark can be statistically valid while the authorship conclusion drawn from it is false. Proofreading, translation, reformatting, or summarization can introduce green-token patterns into material originally written by a person. Model processing is not the same as sole authorship. A detected mark indicates content “may have been processed by Claude,” Anthropic states.

Mixed-source documents make this distinction especially important for content teams. An AI-generated passage can contribute enough excess green tokens to push the whole-document score above threshold even when most of the document is human-written. The resulting flag does not locate the passage or measure its share of authorship. On the other hand, enough human text can dilute a watermarked segment until the score for the whole document no longer triggers detection.

The Geometric Cover Detector and Adaptive Online Locator approaches address the failure of document-level detection in mixed-source material. Their premise distinguishes finding a signal somewhere in a document from testing the document as a single unit. Treating a whole-document flag as an attribution for every sentence ignores that distinction and turns a limited statistical observation into an unsupported editorial judgment.

How the z-score's public design creates structural evasion opportunities

The public z-score formula exposes the quantity a removal attack seeks to reduce. But knowing the formula alone does not reveal the secret green/red partitions. Detector access can provide a separate source of information. An attacker who can compare confidence scores for candidate changes may infer which changes weaken the signal without receiving explicit token classifications.

Researchers demonstrated this feedback mechanism on LLAMA-2 with KGW by querying a detection API with candidate replacements. The attack reduced scores from well above threshold to well below it while maintaining sentence quality. Spoofing reverses the objective by accumulating green tokens to trigger detection on human-written content. BIRA, the Bias-Inversion Rewriting Attack, reported evasion rates exceeding 99% across diverse watermarking schemes in its 2025 findings, using a query-free method while preserving semantic fidelity better than prior baselines.

These results need a precise security interpretation. Small context windows can produce a mismatch between the actual green fraction and preset γ, making detection unstable and patterns easier to learn. Christ et al.’s theoretical result ties computational indistinguishability to secrecy of the key. Public detection alone therefore does not establish vulnerability. For practitioners, the relevant consequence is that a below-threshold score may reflect watermark removal.

What a z-score cannot tell you, and what that means for content workflows

A z-score shows how far the observed green count is from what chance would predict under the detector’s assumptions. It is not the probability that a person or a model wrote the passage. A high score supports a watermark pattern under the tested key, but does not independently establish sole authorship, model version, or generation time. The statistic also cannot show a chain of custody.

A low score leaves several explanations open. The text may be human-written, generated without watermarking, too short or constrained to yield a detectable signal, or altered enough to remove one. In the meaning-preserving paraphrase research described in the brief, every case that was detected before paraphrasing went undetected afterward. That result is bounded by the tested cases, but it illustrates why a detector result alone is insufficient forensic provenance evidence.

For content teams, the practical lesson is to treat the score as one input. A pipeline spanning models, agencies, and tools requires more than one key tested against one passage at one moment. C2PA Specification v2.3, dated January 2026, added support for manifests in unstructured text, including LLM outputs. These layers serve different roles. A watermark can remain when provenance metadata is stripped, while a C2PA manifest supplies identity, timestamp, and provenance-chain information that the z-score cannot provide.

Why single-model pipelines are structurally exposed to z-score detection

Repeated prompts with similar contexts can reuse the same green/red partitions in a single-model pipeline. These repeated-context patterns make the watermark more learnable because the same keyed rules govern token preferences each time. Changing the appearance of the finished text does not necessarily change the mechanism that selected its tokens.

When one model uses the same key across documents, each output can accumulate bias under the same pseudorandom partition rules. More documents can supply more evidence of that consistent bias. Even then, a detector may miss some documents when testing them on their own, since length, register, and editing still affect each result. It does explain why a corpus from one keyed generation process can expose a recurring statistical pattern.

Paraphrasing through the same watermarked model does not reliably break that alignment because the partition rules apply again during rewriting. Architectural diversity changes the underlying token-choice process instead. Models using different keys and partition functions need not favor the same tokens at the same positions, so distributing generation can disperse the excess seen by any one detector. That mechanism offers a reason to investigate multi-model generation, but it does not by itself establish that every such pipeline stays below threshold. The resulting text still requires measurement.

How multi-model generation changes the z-score a detector sees

When contributing models use different watermark keys and partition rules, a token favored by one scheme need not be favored by another. Mixing their contributions at sentence or line level can therefore weaken the systematic surplus visible under any single key. If fewer eligible positions contribute a particular key’s excess, its numerator shrinks relative to a passage generated entirely under that key.

This differs from rewriting through the original watermarked model, which can reapply the same bias. However, dispersal is not a mathematical guarantee of a below-threshold result. The length of the passage and the excess that is left still set the score, while mixed-source detection shows that testing each segment is a separate task. A detector may still find a signal in a contributing passage even when the whole-document score is low. The score also cannot prove that more than one model took part.

A final enhancement pass can unify register and style, but its effect on the watermark must be included in the evaluation. Routing that pass through a single watermarked model can reintroduce the alignment that earlier mixing dispersed. The appropriate proof standard is therefore z-score benchmarking against real KGW detectors. The notes give no quantitative benchmark that shows a success rate for multi-model generation across all cases.