Token-Level Rewrite Rate as a Metric for Watermark Evasion Quality

Token-level rewrite rate measures watermark evasion more faithfully than word-change counts because it tracks edits to the sequence on which detection operates. For KGW watermarks, however, it remains a proxy for the decisive quantity, which is the green-token proportion reflected in the z-score. Evaluating evasion quality therefore requires token-level measurement, attention to each token’s watermark contribution, and a separate test of whether the rewritten text preserves meaning.

How KGW watermarks encode a detectable signal into generated text

KGW watermarking biases token sampling during generation. Green-list token logits receive an additive bonus δ, which increases their sampling probability without forcing their selection. The model still has room to choose. Because the vocabulary partition changes with context, a token can be green in one position and red in another. The detectable pattern therefore resides in the distribution of context-dependent token choices.

Detection reconstructs those green lists and applies a one-proportion z-test. The z-score is the observed green count minus the expected green count, divided by a standard deviation that scales with √T. The signal depends on three parameters, which are the green-list fraction γ, the logit bias δ, and the usable token length T. Longer texts provide more samples for the test. A watermark decision occurs when the excess of green tokens becomes statistically improbable under the hypothesis that the text was not watermarked, but the score itself remains continuous.

Why the z-score, not word-change count, is the correct evasion target

A synonym substitution does not necessarily reduce that excess. If the preceding context remains unchanged, the replacement token can belong to the same green list as the original. Changing context can also change which lists later tokens belong to, so a small-looking edit can affect more than the position that was rewritten. Whether the affected tokens fall onto their new green or red lists is not controlled by the visible size of the edit.

That asymmetry breaks the link between editorial effort and evasion success. A rewriter can change many words and still leave enough green-token hits to keep the z-score above the detection threshold. Small edits can also have disproportionately large effects. Because list assignments depend on context, treating each changed word as an independent contribution to watermark removal gives unwarranted confidence. A useful evasion metric must therefore work on the token sequence in its order, while the detector’s z-score still tests whether the statistical signature survived.

What token-level rewrite rate actually measures and how it is defined

The token editing rate ER_T is the Levenshtein edit distance between the original and rewritten token sequences, divided by the total number of tokens in the original text. It measures the token edits required to transform one ordered sequence into the other. Order is essential to this definition. A bag-of-words comparison would discard the positional relationships that determine the context for green-list assignment. A high ER_T shows that much of the original sequence has changed, though it does not show which edits removed green-token hits.

Character-level distance measures how much the characters have changed. When a single character within a token is modified, both edit distances can be 1, but their normalized rates differ because their denominators differ. Token-level measurement compares the text in the same units the detector uses. It also captures why a visually modest change can have a larger statistical effect than its character count suggests. The relevant disturbance is to token identity and position.

The adaptive optimization described in the research material increased mean token edit distance while minimizing n-gram overlap, and evasion improved as the sequence changed. The process optimized disruption of the original token sequence. Nevertheless, the causal mechanism is the resulting change in green-token assignments and proportions. ER_T is useful because it measures where that disruption can occur, while the z-score establishes its actual effect.

How the leading evasion attacks operationalize token-level rewrite rate

Self-Information Rewrite Attack, or SIRA, concentrates rewriting at high-entropy positions where the original model had genuine choice and green-list bias could influence selection. It leaves low-entropy positions relatively undisturbed because their token choices were nearly forced. Bias Inversion Rewriting Attack, or BIRA, applies a negative logit bias to a proxy suppression set identified through token self-information. This targets likely green-token sampling more directly than ordinary word substitution.

The POSTECH BIRA paper grounds this approach in the result that even a small reduction in average conditional green-token sampling probability can make detection probability decay exponentially. The paper evaluated seven schemes, including KGW, Unigram, UPV, EWD, DIP, SIR, and EXP, using 500 attacked texts calibrated against 500 human-written texts at thresholds corresponding to two false-positive rates. BIRA attacks succeeded more than 99% of the time across those schemes. On a KGW-watermarked sample, the z-score fell from 6.03 to 0.83, while SIRA achieved roughly half BIRA’s average success rate. These results favor targeted statistical disruption.

After one hop, detection fell to between roughly one-seventh and two-fifths of outputs, depending on rewriter and style. After five hops, the strongest settings removed the watermark decision from the large majority of outputs. Character-level perturbation results sharpen the same point, reporting better attack success and watermark-score reduction than token-level attacks at the evaluated editing rates. Small character changes can alter emitted tokens or token boundaries, so fewer visible edits need not mean less token-level disruption.

The quality-evasion tradeoff that token-level rewrite rate also governs

More displacement creates more opportunities to weaken a watermark, but it also exposes meaning, fluency, and style to change. A higher ER_T is not automatically better. The tension is especially clear in low-entropy contexts, where forcing a target green-token hit rate can require large logit shifts because the model assigns little probability mass to the target list. Such shifts can affect factuality as well as phrasing. Rewriting must therefore be evaluated against the content it was supposed to preserve.

Rigorous benchmarks pair evasion with semantic fidelity, paraphrasing strength, and fluency. Semantic evaluation can include GPT-4o judging meaning preservation on a 1-to-5 scale, while BLEU, BERTScore, and perplexity provide narrower statistical views. None is a complete measure of writing quality. A result that appears more human to a detector can still be worse content. BIRA’s reported advantage includes better semantic fidelity than prior baselines, which makes its evasion results more useful than attack success alone.

For content teams, the practical lesson is to pair token-level rewrite rate with a quality gate. The useful operating range is enough disruption to move the z-score below the threshold without losing the intended meaning. Maximum rewriting is not the objective here. A benchmark that reports only successful evasion cannot show whether the output is still fit to publish. Pairing evasion rate with semantic fidelity makes that constraint explicit rather than leaving it to an informal reading after the attack.

Where KGW's own structure creates low-strength tokens that distort rewrite rate readings

KGW produces a substantial share of tokens with near-zero watermark strength, which complicates any uniform reading of ER_T. Low-entropy positions give the model little choice, so a green-list bonus has little sway over which token it produces. Their displacement contributes less to removing the embedded signal. Rewriting many such positions can produce a high edit rate with only modest z-score reduction. Rewriting fewer high-entropy positions can produce the opposite result because those positions provided more room for the watermark bias to act.

SIRA’s selectivity illustrates this unequal contribution within a text. Its concentration on high-entropy positions effectively weights the rewrite budget toward places where the watermark had leverage. Flat ER_T does not express that distinction. Two rewrites with the same edit rate can consequently disturb very different amounts of the original statistical signal. Interpreting the metric requires attention to token contribution as well as the number of changed positions.

A watermarking framework presented at EMNLP 2025 addresses this weakness through part-of-speech entropy, excluding low-entropy tokens from logit perturbations. The framework improves semantic fidelity over KGW while maintaining detectability by concentrating the signal in higher-choice positions. That changes the target for evasion analysis. A more concentrated watermark makes indiscriminate edit counts less informative about how much signal remains. As watermark placement becomes more selective, entropy-weighted token-level rewrite rate becomes an increasingly important refinement of the base metric.

Why multi-model pipelines change what token-level rewrite rate means for enterprise content teams

Multi-stage content pipelines turn this measurement problem into a provenance problem. A researcher, drafter, critic, and finalizer can each operate through a different model, potentially under different watermark regimes or none at all. Each transition can regenerate the token sequence. The next model’s choices displace the context-dependent pattern established under the first model’s key, whether or not watermark removal was the purpose.

The Chainwash study found that detection dropped sharply after rewriting, with the result depending on rewriter and style. Across repeated transitions, an original watermark can become ineffective even though the content still derives from AI generation. Lost detectability exposes a gap between identifying a surviving statistical signature and demonstrating which model produced a particular passage. The final document alone may no longer support that attribution.

Token-level watermarks are statistical signals. They do not provide the equivalent of C2PA’s tamper-evident metadata layer, so a low z-score cannot by itself demonstrate compliance. A replayable run record captures the prompt template version, model used, tool calls, resulting diff, approvals, and a rollback handle, although most platforms lack this record. Rewrite rate reveals how the original signal can dissolve across stages, while the run record addresses the attribution that detection cannot reliably recover.

Why architectural multi-model dispersal addresses the structural vulnerability that rewrite rate exposes

Cross-provider token dispersal addresses statistical concentration during authorship. In a single-model watermarked pipeline, the same key structure repeatedly influences token selection, allowing its green-list excess to accumulate over length. Genuine multi-model authorship distributes contributions across incompatible key structures. When different models contribute candidates in parallel and selection spans providers, the resulting sequence need not preserve one provider’s pattern throughout. This is a structural alternative to relying on later edits to disturb an already established signal.

The attack benchmarks explain the rationale, but they do not establish a benchmark for the architecture itself. BIRA demonstrates the effect of targeted suppression, while repeated rewriting demonstrates how model transitions can erode a watermark. Neither shows what will happen for every multi-model output. Architectural dispersal uses the same mechanism earlier, so there is less need for a post-hoc rewrite whose changes to tokens and writing quality must then be checked.

The strongest quality objection is that diverse authorship can produce an inconsistent voice. Token-level selection with a judge panel and a final enhance pass addresses that concern through editorial selection. Diversity need not remain visible in the prose. Semantic fidelity, fluency, and on-brand voice are evaluated on the unified output. The architectural implication of ER_T is therefore precise, because distributing token preferences at generation addresses the same statistical concentration that rewriting later tries to break, while provenance remains a separate recordkeeping obligation.