I Built a SynthID Watermark DetectorThat Cannot Detect Gemini

The question landed the way good questions usually do, casually, in a chat thread. AI watermarking was suddenly everywhere. Google had been quietly stamping Gemini output with something called SynthID for a while. Anthropic had just announced it was doing the same for Claude. The EU had started requiring AI-generated content to be marked. And there was an obvious product idea sitting right there: build a free tool where someone pastes text and finds out whether an AI wrote it, verified by the watermark rather than guessed at.

It sounded very buildable. Google had open-sourced a reference implementation. There was a peer-reviewed Nature paper. Hugging Face had shipped a production port into Transformers. All the pieces were public.

So I started reading, then building. What I ended up with was a working detector that scores watermarked text at ROC-AUC 1.0000, a benchmark of nearly 4,000 samples, and a conclusion I did not want: the tool cannot be shipped. Not because my implementation was bad. Because the thing it needs does not exist outside two companies, and never will.

I am writing this up because I could not find a single honest account of it anywhere. What I found instead was a search results page full of tools claiming to detect SynthID that demonstrably do not. If you are evaluating this idea, I would like to save you the week.

One thing worth separating up front, because almost every article I read blurred it. Watermark detection and AI detection are completely different mechanisms. AI detection looks at the writing itself, statistically, and guesses. Watermark detection reads a deliberate signal planted at generation time, and either verifies it or does not. This piece is entirely about the second one.

When a language model writes, it does not pick one inevitable next word. At each step it has a ranked cloud of plausible candidates, and it samples one. Often several candidates are near-equally good. "The results were surprising" and "The results were unexpected" are both fine. That freedom is unavoidable, and it is where SynthID lives.

Here is the mechanism, in the order it happens:

Take the previous four tokens as context. For each candidate next token, hash the pair together with a secret key. That hash gets mapped to a single bit, either 0 or 1, called a g-value. Since the hash is deterministic, the same context plus the same candidate plus the same key always gives the same bit. Now run a small knockout tournament between candidates, preferring those whose g-value is 1. Repeat this thirty times over, once per key, in thirty stacked layers.

The word that wins is still a perfectly good word. The text reads normally, because the model was genuinely indifferent between the options. But the winner is now slightly more likely to be one whose g-value was 1.

Do that for one token and you have learned nothing. Do it for four hundred and you have a measurable statistical bias. On unwatermarked text, the average g-value sits at exactly 0.5, a fair coin. On watermarked text it drifts upward. That drift is the entire watermark.

I find the coin-flipping analogy the clearest one. Imagine a rule that says: whenever you have a free choice, flip a keyed coin and let it break the tie. Anyone holding the same coin can replay every flip and notice you were winning far more often than chance allows. Anyone without the coin sees a person making ordinary, unremarkable choices.

Two consequences fall straight out of this design, and both matter enormously later.

The watermark needs the model to be uncertain. If one word has probability 0.999, every tournament branch picks it anyway, g-value or not. So high-entropy creative prose carries a strong watermark, while factual writing, code, and low-temperature output carry a weak one. Anthropic says this plainly in its own documentation: watermarking is sparser on factual passages where fewer choices can be made without hurting accuracy.

The watermark is a correlation, not a payload. There is no hidden data to extract. There is only a statistical relationship between the text and a specific key. Hold that thought.

Both implementations, Google's reference repo and the Hugging Face production port, derive an identical 64-bit hash from the same inputs. Same keys, same context, same candidate token, same linear congruential generator, same constants. Then they turn that hash into a g-value in two completely different ways.

The reference repo re-applies its hash function twelve more times, shifts the result, and takes a single bit out of position 30. The Transformers port takes a pre-generated table of 65,536 random bits and indexes into it using the hash modulo the table size.

Both are perfectly valid ways to produce a fair pseudorandom bit. Neither is wrong. But they are different functions, so they produce different g-values from the same text.

I wrote a test comparing them across thousands of tokens. They agreed 45 to 55 percent of the time, which is to say, exactly as often as two unrelated coins agree. Chance.

The practical implication is sharp: a watermark applied by one implementation is completely invisible to a detector built on the other, even with identical keys, identical n-gram length, identical everything in the config. If I had generated my test data with one library and detected with the other, I would have seen nothing at all, and the obvious conclusion would have been that my code was broken. I might have spent days there.

So a watermarking configuration does not define a watermark. The full specification is the key list, the n-gram length, the tokenizer, the context history size, and the g-value sampler implementation, including the exact random table and the seed that made it. Change any one and detection returns chance.

I turned that into an assertion in my test suite rather than a note in a README, because it is precisely the kind of assumption that rots silently.

There is a second, quieter version of the same trap. The g-values are hashed from token IDs, not characters. Detect with a different tokenizer and you are hashing a different integer sequence entirely, which produces pure noise. There is no partial credit and no warning, just a score near 0.5. If you do not know which model produced a piece of text, you cannot reliably check its watermark even holding the right key.

The trap with any detector is that your positives and negatives usually differ in more ways than the one you care about. If I generate watermarked text with a model and compare it against Wikipedia articles, a high score might mean I detected the watermark, or it might mean I detected "sounds like a language model". Those are very different products.

So the core of the benchmark is a control that removes that ambiguity. For every watermarked sample I generated a twin: same model, same prompt, same random seed, watermark switched off. Two texts that are stylistically indistinguishable, that started from identical sampling noise, differing in exactly one variable.

Then I added human writing from Wikipedia and Project Gutenberg, plus output from a different model entirely, to catch a detector that had accidentally learned a house style.

Across 3,924 samples, at a threshold set for a 1 percent false-positive rate:

ROC-AUC 1.0000. Detection rate 100 percent. Measured false-positive rate 1.2 percent, against a 1 percent target, which is about as well calibrated as a few hundred negatives can demonstrate.

The number I actually care about is the breakdown by negative type. Human text produced a 1.4 percent false-positive rate. The other model produced 0.0 percent. And the same-model, same-prompt, same-seed twins produced 1.0 percent, with a median score of 0.4999 against a chance value of 0.5.

That last figure is the one that makes the detector real. Those samples are the hardest possible negatives, and they land on chance to four decimal places. The detector is reading the watermark, not the writing. No perplexity-based AI detector can make that claim about its own negatives, because it has no way to construct that control.

A couple of other things I checked, because they are the standard ways this kind of work goes wrong:

The theoretical ceiling from the Nature paper, assuming a maximum-entropy model, is a mean g-value of about 0.7500. My watermarked text scored a median of 0.6252, roughly half the available headroom. A score above the ceiling would have meant I was double-counting something.

I also refused to let the detector guess on short text. Below about 25 usable n-grams, no threshold separates the classes at a useful error rate, so it returns "inconclusive" with a reason instead of a verdict. Those refusals are reported separately and never folded into the accuracy figures, because trading a refusal for a coin flip is how detectors get to claim numbers they have not earned.

And I never reported accuracy. Accuracy depends on how many positives happen to be in your test set, so it describes your dataset rather than your detector. Rates, ranking quality, and detection-at-a-fixed-error-rate are the numbers that transfer.

Everything above used the research keys that ship with Google's public reference code. Thirty integers, published so people can reproduce the paper. Real Gemini uses different keys, and those are secret.

I had been assuming this would be a matter of degree. Wrong key, weaker signal, maybe still something detectable if the text were long enough. So I took genuinely watermarked text and detected it with the same keys, incremented by one.

| | score | verdict | | correct keys | 0.6278 | detected, p = 3.26e-26 | | keys off by one | 0.5023 | nothing, p = 0.42 |

Same text. One digit different. The watermark did not weaken. It vanished, landing on chance as cleanly as a human-written paragraph.

That is when the mechanism I described in chapter two stopped being an interesting detail and became the whole answer. The watermark is not concealed information sitting inside the text, waiting for someone clever enough to notice it. It is a correlation between word choices and a keyed pseudorandom function, and that correlation only exists relative to the key. With the right key, bias. With any other key, fair coins.

SynthID is specifically engineered to be distortion-free: the statistical distribution of the text is provably unchanged by watermarking. That is a feature, so watermarking does not degrade output quality. It is also a wall. There is no entropy artefact, no perplexity anomaly, no fingerprint of any kind to find, because by construction there is nothing there until the key makes it appear.

I went looking for a way around it anyway. Every route dead-ends on the same property:

Brute-force the keys. Even assuming the small integer range Google uses in its research config, that is on the order of 10^90 combinations. And you would still need their exact tokenizer, depth, n-gram length and sampler variant, none of which are published.

Extract the keys from the model. They live in the provider's server-side sampling loop. They are not in the weights and never reach the client. For a closed model there is nothing local to pull them out of.

Steal the watermark statistically. This is the serious candidate, and researchers at ETH Zurich have done it. Bulk-query the model, learn an approximation. Against SynthID it achieved 4 percent spoofing success at baseline budget, rising to 15 percent at 90,000 queries, far more resistant than other schemes which collapse above 80 percent. But it detects whether a model watermarks, across a corpus, not whether one document does. It also needs an unwatermarked reference from the same model to compare against, which does not exist when every response from that model is watermarked.

Detect side-effects. Ruled out by the distortion-free property. And anything you did find would be generic AI detection wearing a costume.

Read provenance metadata. Works for images and files. Text pasted into a box has none.

So the honest answer to "can we detect it, ignoring which company made it" is no, and dropping the attribution requirement does not help at all. Presence and readability are the same problem here. You cannot tell that an envelope is sealed without the key that opens it.

This is not an obscure limitation, either. It has a name in the literature, public verifiability, and it is an active research problem. The dilemma is stated cleanly in a USENIX Security 2026 paper: the key cannot be public, or an adversary can strip and forge watermarks at will; but it cannot be private, or detection is opaque to everyone else. Proposed solutions involve building zero-knowledge proofs into the watermarking scheme itself. That requires the provider to adopt new machinery. It is structurally not solvable from outside.

I ran every sample through a set of deterministic transformations, applied to positives and negatives alike so that each transformation has its own false-positive rate rather than borrowing one from unedited text. Detection rates below are at a 1 percent false-positive threshold.

Vocabulary does not matter. Word order does.

Replacing every single word that appeared in my synonym table left detection at 100 percent. Untouched. Meanwhile, swapping 20 percent of adjacent word pairs, a much smaller-looking edit, dropped it to 85 percent, and stacking edits together took it to 48 percent.

The reason is structural. Each hash covers a five-token window. Displace one word and you damage up to five overlapping windows, not one. So an edit that preserves sequence is nearly free, while an edit that disturbs adjacency is expensive. Swapping synonyms one-for-one keeps every window's shape intact. Reordering shreds it.

Making text shorter does not help.

I expected truncation to be the easy attack. It is not. Cutting text to a quarter of its length kept detection at 100 percent. The score is a rate, an average over positions, not a total that accumulates. Removing half the text removes half the evidence and half the positions, and the average sits exactly where it was.

What shortening does change is confidence. Evidence grows with the square root of token count, so a short passage becomes genuinely undecidable rather than falsely clean. Which is a much better failure mode for a detector than it sounds.

Low-entropy text is fragile in every direction.

I ran the whole benchmark twice, once at temperature 1.0 and once at 0.5. At the lower temperature the watermark starts weaker, a median of 0.5579 against 0.6252, roughly half the signal, simply because the model had fewer free choices to encode into.

And every attack then became dramatically more effective. Adjacent word swaps went from costing 15 points to costing 70. Heavy rewriting went from 48 percent detection to exactly zero, with ranking quality collapsing to 0.60, barely above a coin toss.

So robustness is not a property of the watermark alone. It is a property of the watermark and the entropy it was hidden in. Factual writing, technical explanation, anything constrained: weaker watermark, easier removal.

This lines up with what everyone else who has measured it reports. ETH Zurich found SynthID easier to scrub than comparable schemes, with ordinary paraphrasers achieving over 90 percent removal. Anthropic states it directly: light editing probably will not remove the watermark completely, but a rewrite where every word is replaced will. Brookings reaches the same conclusion from a policy angle, noting that modest paraphrasing can erase the signal and that watermarking is therefore of limited use in high-stakes settings.

Which raises the uncomfortable practical point. Even if you could detect these watermarks, the texts most likely to be edited before submission are exactly the ones you would most want to catch.

Here is the thing I had to sit with. A public watermark checker built on this would be technically flawless and completely useless. For ChatGPT text there is no watermark to find. For Gemini and Claude text there is one, but we cannot read it. So the output would be "no watermark detected" for every single real submission, forever. Not inaccurate. Just a constant function wearing a lab coat.

And the failure mode is worse than useless, because people would read "no watermark detected" as "not AI-generated". The tool would be right and the user would be misinformed. That is the exact trap the existing tools fell into, and I did not want to add to the pile.

So what was the point?

First, a defensible negative result. "We cannot build this" is a real deliverable when it is backed by an implementation rather than a hunch. I can now explain precisely why, cite the mechanism, and produce the one-digit key experiment on request. That is a different thing from reading a few blog posts and guessing, and it is the reason I trust it enough to write this. Getting to a confident no required building the thing.

Second, and more usefully, a measuring instrument. Every AI-detector benchmark I have run has the same weakness: you are scoring against someone else's black box. When a number moves you cannot tell whether your text changed or their model did. Ground truth is borrowed.

Here, I control the watermark completely. I know exactly which texts carry it and exactly how strong it is. That makes it possible to ask a question you cannot otherwise ask cleanly: how much statistical signal does a given rewriting process actually remove? Not "did a third-party detector's score go down", but a measured quantity against known ground truth. Feed in watermarked text, run it through any rewriting pipeline, measure what is left. That harness works, and it outlived the product idea that motivated it.

Third, transferable knowledge. The fragility findings do not depend on whose key was used, because the mechanism is identical either way. Word order matters and vocabulary does not. Truncation is not an attack. Low-entropy text is weakly marked. Those hold for Gemini and Claude too, and they are the kind of thing worth knowing before you form an opinion about whether watermarking will work as a policy instrument.

The reproducible parts, for anyone who wants them: 95 tests, the closed-form false-positive threshold validated against Monte Carlo, mask computation verified byte-for-byte against Google's reference code, the two-implementation divergence asserted rather than assumed, and a wrong-keys test that fails loudly if the detector ever stops being key-dependent.

If I am honest about the ledger: a week spent, a working detector, a benchmark I still use, and a product idea correctly killed. I would rather kill an idea in week one with evidence than in month six with a support inbox.