Invisible Unicode characters in AI-generated text

01What counts as an invisible character

Unicode reserves several ranges for characters that carry instructions rather than shapes. The zero-width space (U+200B), zero-width non-joiner (U+200C) and zero-width joiner (U+200D) control how neighbouring glyphs bind together. The byte order mark (U+FEFF) survives from an era when it signalled a file's endianness. Bidirectional controls (U+202A to U+202E) switch text direction mid-string for mixed Arabic, Hebrew and Latin content. The soft hyphen (U+00AD) marks a place a word may break across lines and otherwise shows nothing.

Alongside these sit space characters that render as whitespace but are not the ordinary U+0020: the narrow no-break space (U+202F), the ideographic space (U+3000) used in CJK typesetting, and the en quad through hair space range (U+2000 to U+200A) inherited from metal typesetting.

None of these are visible in a browser, an editor, or a word processor. All of them are ordinary text as far as the clipboard is concerned.

02Why they appear in model output

The usual cause is inheritance rather than intent. Models are trained on text scraped from the web, and the web is full of these characters — pasted from PDFs, exported from design tools, copied out of rich-text editors that use zero-width spaces to control line breaking. A model that has seen them will occasionally emit them.

The second cause is the rendering layer rather than the model. Chat interfaces format responses as HTML and apply typographic niceties: a narrow no-break space around an em dash, a soft hyphen in a long word. Copying from the rendered page picks these up even though the model never produced them.

Neither mechanism is a watermark. A watermark is a signal placed deliberately so it can be detected later; these are artefacts.

03How to tell whether text contains them

The simplest check is a character count that disagrees with what you can see. Paste the text into any tool that reports length and compare with a manual count of a short sample.

In an editor, a regular expression will find them directly. The range that matters most in practice is /[\u200B-\u200D\uFEFF\u00AD\u2060-\u2064]/g, though a thorough sweep covers considerably more, including the Unicode tag block described in the companion page on smuggled text.

Some editors can render them. VS Code highlights unusual invisible characters by default and will show a marker where one sits.

04What removing them accomplishes, and what it does not

Removing them fixes real problems. Invisible characters break exact-match search, cause string comparisons to fail for text that looks identical, corrupt CSV and JSON parsing, and produce phantom diffs in version control. A few naive AI-detection tools also flag their presence, so stripping them removes a weak signal.

It does not remove a statistical watermark. That mark lives in which words the model selected, not in extra characters, so deleting characters leaves it entirely intact. Any tool that claims character removal defeats a token-level watermark is describing one mechanism and selling another.

One caution: the zero-width joiner is load-bearing inside emoji. Removing every U+200D indiscriminately turns a family emoji into four separate people. A careful cleaner preserves the joiner when it sits between two pictographs.

Questions