Evaluating AI Writing Tool Claims Without Independent Benchmarks

Content teams need to know enough about how the tools work to turn claims about detection evasion, quality, and compliance into tests. The governing principle is straightforward. A claim earns trust only when its detector, test conditions, and results are explicit enough for someone else to challenge.

Why vendor claims about AI writing tools outpace what can be independently verified

A content team is told that a vendor’s output is “undetectable,” but receives no z-score, no detector name, and no specified text length. There is nothing concrete to test. The missing conditions prevent the team from distinguishing a narrow result from a general promise. No independent body systematically benchmarks these claims against real detectors or documented regulatory requirements, leaving buyers to establish the scope of the evidence themselves.

That gap extends beyond detection to quality superiority and compliance readiness. The European Commission’s official FAQ says that EU AI Act Article 50 applies from 2 August 2026, so buying decisions need to take into account when transparency obligations start and what they cover. Detection resistance is not evidence of legal compliance. Meanwhile, watermarking mechanisms, detection algorithms, and evasion research evolve in parallel, so a launch-time result may become obsolete within months. Teams accepting an unqualified promise inherit the risk that its unstated assumptions no longer hold.

How KGW watermarks are embedded and what the z-score actually measures

KGW is the dominant academic approach to text watermarking. Before each token is sampled, the token just generated seeds a hash function that uses a secret key to split the vocabulary into green and red lists, roughly half each. A small constant bias δ is added to the logit scores of green-list tokens. This makes those tokens more likely to be selected without requiring any particular one to appear. The watermark resides in the accumulated preference.

Using the same key and hash function, a detector re-derives each position’s vocabulary partition and counts the green tokens. A one-proportion z-test measures how far their observed share departs from the chance baseline. Zhao et al. (ICLR 2025) describe the z-score threshold as the operative criterion for infringement detection, using the same statistical logic relevant to content detection. The score tests only that signal and says nothing about writing quality or legal status.

For a sustained excess of green tokens, the z-score grows with the square root of document length. Short passages therefore rarely accumulate enough evidence to reach significance, while longer passages can expose the same underlying bias. A resistance claim without text length omits a central test condition. Because the bias enters at the sampling layer rather than through the model’s instructions, prompting cannot simply switch it off.

The watermarking family is also changing. Entropy-based Watermark Detection (EWD) reweights tokens by entropy to improve the detectability and quality tradeoff, while Semantic-Invariant and Robust (SIR) watermarking derives its green list from a semantic embedding of the prefix. Hou et al. show that k-SEMSTAMP, published at ACL Findings 2024, uses k-means clustering of semantic space to improve robustness against paraphrase attacks over LSH-based SEMSTAMP. These variants change what a detector can preserve through rewriting. SynthID represents an alternative to the KGW mechanism.

What watermark evasion actually requires, and why surface-level rewriting is not the answer

A KGW watermark is disrupted by changing the token choices that produce the green-list excess. Light paraphrasing changes only part of that evidence. It can lower the z-score and still leave enough of the original sampling pattern for detection, especially if the passage is still long. The vendor pitch that treats a cosmetic rewrite as dependable removal therefore exceeds what the mechanism supports. A lower score is not necessarily a score below the detection threshold.

Re-composition changes the relationship to the source more substantially. A model generating from a summary or outline replaces the original token sequence. That removes the original choices, although the new generation still has its own statistical properties. Translation-based attacks also break up the source pattern because a different tokenizer uses different vocabulary partitions. Neither mechanism establishes that every resulting document will escape every detector.

The stronger objection is that sufficiently aggressive paraphrasing can work. However, changes extensive enough to disrupt detection can also alter meaning and voice, exchanging a detection problem for a quality problem. Semantic watermarking variants further weaken the assumption that lexical changes alone are sufficient. The practical verdict is therefore narrower than either “paraphrasing works” or “paraphrasing never works.” Light post-processing is a fragile basis for a general promise, and deeper rewriting must be judged against the content it preserves.

How a single model's token distribution creates a persistent, identifiable fingerprint across documents

Watermarks are deliberate signals, but fingerprints need not be. Ghosal et al.’s survey distinguishes prepared detectors, which require watermarking at generation time, from post-hoc detectors, which classify text using its statistical or stylometric properties. The latter need no embedded green-list bias. A team can therefore avoid intentionally watermarked models and still produce text with detectable regularities. Removing one source of evidence does not remove the other.

Intrinsic fingerprints arise from a model’s architecture, training, and sampling behavior. They differ from injected fingerprints, such as deliberately embedded trigger-target pairs, because they require no coordination with the model owner. Repeated use exposes these regularities across documents. When one model produces thousands of articles, corpus-level repetition can make its signature more robust than any isolated passage suggests. This is why a polished sample is insufficient evidence about a production pipeline.

A lightly fine-tuned open-weight model may keep traits that can be traced to its base, even when it is sold as proprietary. LIDet achieves over 90% accuracy in distinguishing infringing from clean models, illustrating the reach of model-fingerprinting research rather than establishing a detection rate for individual articles. The distinction is important to interpreting that figure. A single-model architecture supplies a recurring distribution for detectors to examine, but it does not establish that every document will be detected.

Why multi-model architectures address the fingerprint problem structurally, not cosmetically

Dispersing authorship within a document across models from different providers means the text no longer comes entirely from one sampling distribution. This changes how the text is written, before a statistically consistent passage exists for another model to polish. It directly addresses the concentration of authorship that makes single-model fingerprints persistent.

For KGW-style watermarking, the relevant distinction is between coherent and dispersed bias. A detector reconstructs partitions using a particular key and hash function, while different models may contribute text shaped by different keys and sampling distributions. Mixing those contributions can weaken the evidence accumulated under any one partition. However, architectural dispersal is not itself a measured z-score. Whether the signal falls below a threshold depends on the detector, text length, and actual blend being tested.

The level of mixing is therefore consequential. Routing whole documents to different models leaves each document with its own single-model signature, whereas token- or line-level dispersal changes authorship within the unit being tested. Task routing can also improve quality and reduce inference cost, but those benefits do not establish detection resistance. Being harder to detect also does not prove that the text meets disclosure obligations. Multi-model composition is a structural response to concentrated fingerprints, with an unqualified “undetectable” label requiring evidence beyond the architecture itself.

Building your own evaluation criteria when vendors won't publish theirs

For content teams, these mechanisms translate into a practical evaluation protocol. Begin by asking which detector supports the claim, at what text length, and with what z-score threshold. A vague answer leaves the claim unfalsifiable. Test at production lengths rather than the vendor’s demonstration length, because output that appears resistant at 300 tokens may cross the threshold at 800. The relevant test unit is the content the team actually publishes.

Next, run a KGW-family test with the necessary key and documented parameters. The algorithm is published, and a data-literate team member can apply the one-proportion test when the text and matching key are available. Key access is a real prerequisite. A test with an unrelated key does not evaluate the original keyed watermark. Ghosal et al.’s survey provides the broader conceptual map for separating prepared detection tests from post-hoc methods, so that one successful test is not mistaken for comprehensive coverage.

Record the raw output’s z-score, then the score after a light paraphrase pass and after a structured multi-model blend. The differences reveal what each intervention changes. Tests at the lengths the team uses show when the signal weakens by different amounts, which a single pass result can hide. Aggressive transformations also need separate checks for changes to meaning and voice, because detection performance alone cannot show whether the quality is good enough to use.

Fingerprint testing requires a corpus. Generate documents through the vendor’s tool and check how much their token-distribution patterns share the same statistical traits. High cross-document similarity is a fingerprint signal. Separately, run the same brief multiple times and compare voice consistency, since uncontrolled routing can produce variation that undermines editorial use. Ask the vendor for its own z-score benchmarks with methodology attached, then compare those conditions with the team’s protocol.

What a credible vendor benchmark looks like, and how to read one honestly

A credible benchmark exposes its claim to disproof. For a KGW-family claim, it names the detector, specifies the secret key or key-generation method, gives the document length and z-score threshold, and reports the actual scores. Pass or fail alone conceals the distance from the boundary. The methodology must also contain enough detail for an external evaluator to repeat the test. If the team’s technical staff cannot reproduce it, the benchmark cannot serve as dependable compliance evidence.

Short passages can take advantage of how little evidence they give for a significant z-score, while weak or outdated detector versions take advantage of the lag between research and the detectors in use. Cherry-picked content types can make performance look broader than it is. Zhao et al. show that the choice of queries makes a significant difference to detectability in tests of whether a suspect model was trained on watermarked data. That finding concerns infringement detection, but it reinforces the need to test an evasion claim across the prompt types and domains in which it will be used.

Detection tools require a corresponding standard of evidence. Their benchmarks should report false-positive rates at specified thresholds and text lengths, rather than treating every flagged document as correctly identified. Human-written text belongs in that evaluation. A detector that wrongly labels it can create liability rather than reduce it, and a low score on one evasion test says nothing about that error rate. Prepared watermark detection and post-hoc classification must therefore remain distinct when interpreting results.

The minimum bar for a KGW watermark-evasion claim is a real detector test with reported z-scores and reproducible methodology. Broader promises need more evidence than that narrow result, with separate tests of quality, fingerprint risk, and compliance. Missing parameters are diagnostic information about a claim’s maturity. Content teams need not wait for a universal benchmark to reject claims that cannot be tested, but they should apply the same discipline to architecture-based promises as to paraphrasing pitches. The standard is evidence under stated conditions.