Reproducible Benchmarking Methodology for AI Watermark Removal Tools

Reproducible benchmarking of AI watermark removal tools must connect each output to a real detector’s decision threshold. For KGW-family detection, that means reporting the z-score before and after processing, under a disclosed configuration. Neither establishes whether a particular document crossed the boundary that defines detection.

Why KGW detection is built on a z-score, not a verdict

KGW belongs to the family of green-list watermarking schemes. At each generation step, a hash of the preceding token and a secret key deterministically divides the vocabulary into green and red lists. The generator adds the same bias δ to each green-list logit before sampling. This changes token probabilities.

Detection reconstructs the relevant green lists and counts eligible tokens that fall within them. It treats each scored token as an approximately independent Bernoulli trial, with a success probability equal to the expected green-list fraction. Under the null hypothesis of no watermark, the observed count should fluctuate around that expectation. More green tokens than expected are evidence against the null. The detector measures that excess rather than judging whether the prose sounds artificial.

The resulting statistic is z = (x − np) / √(np(1−p)), where x is the observed green-token count, n is the number of eligible tokens, and p is the expected green fraction. The detector uses a standard one-sided z-test. Its output is continuous. For a fixed excess green-token fraction, z grows with the square root of document length, so short texts have less opportunity to accumulate significant evidence.

What a detection threshold actually means, and why crossing it is the only thing that matters

The detector turns that continuous statistic into a decision by applying a fixed significance threshold. A text is flagged when its z-score exceeds the threshold, while a score below it passes that detector’s test. The threshold defines the operational boundary. A benchmark therefore needs both the score and the boundary against which it was evaluated to substantiate a label such as “removed.

A tool that reduces z from 15 to 5 has not cleared a detector whose threshold is 4. Reducing it to 3 crosses that boundary. The first reduction is substantial statistically but insufficient operationally. Passing the detector is also a narrower claim than satisfying a legal obligation, which cannot be established by a low z-score alone.

Perplexity, BLEU, chrF, and semantic cosine similarity describe output quality or similarity to the original. Good scores can coexist with continued detection. Quality measurement remains useful alongside the detection result.

True Positive Rate at a fixed rate of incorrect positive detections is a better measure because it shows how well the detector works. TPR/FPR is rigorous at the population level. It describes how the detector behaves across a corpus under a chosen operating condition. A per-output z-score shows where one document sits in relation to that operating boundary. Content teams need that individual result when deciding whether their own output meets a specified detection criterion.

How existing benchmarks measure the wrong thing

Tests often measure how far an attack lowers a detector’s True Positive Rate without sharing the scores for each text behind that change. That demonstrates reduced detection across the tested population. Even when the underlying experiment is rigorous, aggregate results do not locate a particular output relative to the threshold.

The WATERPARK benchmark, introduced in EMNLP 2025 Findings, integrates multiple watermarkers, removal attacks, and metrics in a comprehensive reproducible harness. It evaluates detector accuracy using TPR at a low fixed FPR. That supports controlled comparisons across attacks and schemes. Its aggregate metrics, however, do not expose individual text z-scores or their threshold-crossing behavior.

The PAN 2026 Text Watermarking shared task proposes evaluating inconspicuousness and robustness against unknown obfuscation. Those are legitimate system-level goals. They address how watermarking performs under changes that an evaluator may not know in advance. They do not, by themselves, provide a threshold-anchored result for each document a content team intends to publish. A per-output benchmark adds to this research while leaving its goals in place.

Quality-focused evaluations leave a related gap. KGW implementation studies use measures such as chrF and semantic cosine similarity to describe how much an output changed. Yet near-identical meaning can accompany threshold crossing, while obvious degradation can leave a watermark detectable. Governance proposals also often call for resistance to tampering without defining the required degree of robustness. Without an agreed protocol for acceptable z-score degradation across the content lifecycle, a claim such as “100% conditional removal” cannot establish how a specific document will perform at its own length and configuration.

Why removal claims are unreliable without a threshold-anchored z-score

A short output may remain below threshold because it contains too few eligible tokens to accumulate a significant signal, even before processing. The tool may get credit for a score that was already below threshold. Reporting length together with the original score distinguishes that case from an actual crossing caused by the intervention. Results on short samples therefore cannot be generalized to long-form content without testing the relevant length range.

Scheme choice creates another limit on generality. Unigram uses one fixed global vocabulary split and is described as the most edit-robust of the green-list schemes. An attack that succeeds against KGW need not succeed against Unigram. The scheme name alone is still insufficient. A benchmark must disclose the configuration that determines how strongly the watermark is embedded and how the detector evaluates it.

ChainMark, presented at the Trustworthy AI for Good Workshop at ICML 2026, identifies that KGW’s watermark strength δ is set by hand. That creates a calibration gap for removal claims. Success at one bias setting does not show that a tool will succeed at a stronger setting. ChainMark’s relevance here is configuration transparency. A reproducible claim needs the watermark parameters, detector key, threshold, per-output score distribution, and tested length range.

The removal paradox adds a separate caution. Image-domain research reports that removing a watermark can itself leave detectable traces, illustrating why defeating one signal does not establish the absence of other evidence. The evidence for this effect remains confined to the image domain. For text benchmarks, the defensible claim remains specific to the tested detector and configuration. “Below this threshold” is measurable, while an unrestricted claim of being undetectable reaches beyond what the test establishes.

What a reproducible z-score benchmark requires in practice

To address these failure modes, a valid benchmark must specify the full detection pipeline before interpreting removal results. The scheme and exact configuration come first. For KGW, that includes the green-list ratio and bias δ, while other schemes require their own parameters. A benchmark covering KGW, Unigram, the SynthID family, or Gumbel/exponential methods must preserve those distinctions rather than assume that every detector shares KGW’s statistical interpretation.

The key must also be available for reproduction. Where a vendor retains its production key, the research setup needs a disclosed equivalent key used with an open implementation. That produces a reproducible experiment, but it is not a measurement against the undisclosed production key. The benchmark must state the detection threshold and its corresponding false positive rate. It must also report the corpus’s document-length range so that readers can identify whether the experiment covers their intended output.

For each tested output, the benchmark must report z-scores before and after removal. Those paired measurements expose actual threshold crossings. The results must also distinguish the fraction of texts that began above threshold from the fraction that ended below it. Otherwise, it is hard to tell which samples were already undetected and which passed because of the tool, while results for texts detected at the start can be mistaken for results across all output. Aggregate TPR remains useful as a companion to those records.

Output quality belongs in the same evaluation, measured separately from detection. Coherence and voice consistency can deteriorate even when an output clears the threshold. Conversely, strong semantic preservation does not establish removal. The benchmark therefore needs quality metrics beside the z-scores, so readers can judge the cost of crossing the boundary.

WATERPARK provides a reproducible harness for evaluating watermarking and attacks through open implementations. Separately, open implementations describe reproducing KGW, SynthID, Unigram, Gumbel, SWEET, and EXP-Edit bit-for-bit against their official papers. These are distinct claims. Neither makes an openly keyed research result interchangeable with a vendor’s production result. Reproducibility depends on reporting precisely which implementation, key, and configuration produced the measurements.

An EU regulation’s Article 50, effective August 2, 2026, mandates machine-readable marking of AI-generated content. Anthropic, Google, OpenAI, another technology company, and Microsoft are among the organizations that signed the voluntary transparency code. Threshold-anchored measurements can answer the technical question of detectability under a specified test, but they do not turn watermark removal into proof of legal compliance.

Why single-model pipelines are structurally unable to pass this benchmark

A single-model pipeline with watermarking applied at generation repeatedly introduces bias at the sampling layer. Human editors do not see that layer directly. A prompt to “sound more human” changes the requested style while leaving the watermark configuration that shapes token selection unchanged. Editing the surface of the text therefore gives no guarantee that the final output will fall below threshold.

A single ChatGPT paraphrase pass can substantially weaken a watermark, and published evaluations show sharp TPR reductions under moderate paraphrasing. Residual signal may still accumulate across enough eligible tokens. A single-model pipeline can produce individual outputs that pass a detector, but those successes do not establish reliable clearance across configurations and document lengths.

The objection that published experiments achieve “100% conditional removal” must be read with its condition intact. It concerns texts detected before the attack. WATERPARK reports baseline TPR values of approximately 0.97–1.00 across tested watermarkers on pristine text, followed by sharp declines under attacks. Those high starting rates do not show how far each document was above threshold. Only paired z-scores across a representative length range can distinguish marginal detections from stronger signals and show which ones were actually cleared.

Multi-model authorship blended at the token level addresses a different part of the mechanism. When providers use differently keyed green-list assignments, mixed output need not accumulate the same coherent bias as text generated under one key. KGW struggles when watermarked text is diluted with extensive non-watermarked text. This supports the architectural case for disrupting a consistent signal. It does not make model count a substitute for measurement.

Multi-model output needs a record of per-output z-scores, tested with the specified detector and threshold across the relevant length range. Otherwise, an architectural claim becomes another proxy for the result it is supposed to establish.