AI Desk · Model Identity

Tokenizer Atlas

Explore the tokenizer-family structure underlying the AI model ecosystem: which models read text with the same vocabulary, where the families sit, and which entries share ancestry their labels never mention.

Models measured: 280
Complete fingerprints: 262
Distinct fingerprints: 65
Tokenizer groups: 51
Groups with 2+ models: 26
As at: 18 September 2026

How to read this page: a tokenizer is the fixed vocabulary a model chops text with, and it cannot change without retraining, so it works like a fingerprint. Of the 280 models measured, 262 returned all fifty rows clean; only those carry an exact signature and appear in the groups below. A group gathers the signatures the evidence shows to be one tokenizer.

The constellation

One star cluster per tokenizer group; size follows membership

Tap or hover a cluster for its family; click to jump to its members. Position and distance on the canvas carry no meaning: cluster size follows membership, and only the grouping itself is data. The full listing below carries everything the canvas shows, so nothing depends on it.

The fingerprint matrix

One column per major group, one row per test string. A bright row is a string that tells the vocabularies apart; a faded row is one they nearly agree on.

Each cell is the marginal token cost of that string under that group's exemplar, shaded within its row from the row's lowest count to its highest. Exact values sit in the cell tooltips, and two models are compared string by string on thebench.

Family explorer

One card per inferred tokenizer group with two or more models; unique signatures sit in their own section beneath. Signatures are grouped when the evidence shows one vocabulary behind different serving templates.

Qwen · group Afg_920d45ce · 39 models · 3 measured signatures · 1 inferred tokenizerexact signature tk_5410b38e · 37 models template variant tk_9c4e2396 · 1 model template variant tk_920d45ce · 1 model

and 31 more

GPTfg_d1411c9d · 32 models · 1 measured signature · 1 inferred tokenizer

and 24 more

Mistral · group Afg_e0b4c23e · 24 models · 3 measured signatures · 1 inferred tokenizerexact signature tk_0da8a999 · 22 models template variant tk_b0a747db · 1 model template variant tk_e0b4c23e · 1 model

and 16 more

Gemini · group Afg_a15512c7 · 21 models · 2 measured signatures · 1 inferred tokenizerexact signature tk_59ca250f · 20 models template variant tk_a15512c7 · 1 model

and 13 more

Qwen · group Bfg_c5c0a8ed · 17 models · 2 measured signatures · 1 inferred tokenizerexact signature tk_181907f2 · 15 models template variant tk_c5c0a8ed · 2 models

and 9 more

DeepSeekfg_869dc1cf · 15 models · 2 measured signatures · 1 inferred tokenizerexact signature tk_73c4f0dc · 13 models template variant tk_869dc1cf · 2 models

and 7 more

Llama3fg_066351de · 14 models · 3 measured signatures · 1 inferred tokenizerexact signature tk_6588ed5b · 8 models template variant tk_47615f2d · 5 models template variant tk_066351de · 1 model

and 6 more

Unlabelled groupfg_c018f9f2 · 9 models · 3 measured signatures · 1 inferred tokenizerexact signature tk_133b086c · 7 models template variant tk_1d05586d · 1 model template variant tk_c018f9f2 · 1 model

and 1 more

MiniMaxfg_9f84b5fb · 8 models · 1 measured signature · 1 inferred tokenizer

Z.aifg_3eaaf3e0 · 7 models · 1 measured signature · 1 inferred tokenizer

ByteDance Seedfg_25bf52d0 · 6 models · 1 measured signature · 1 inferred tokenizer

InclusionAIfg_fc5a4585 · 5 models · 1 measured signature · 1 inferred tokenizer

OpenAI cl100kfg_ba918336 · 4 models · 1 measured signature · 1 inferred tokenizer

Tencent · group Afg_2b541395 · 4 models · 1 measured signature · 1 inferred tokenizer

Qwen · group Cfg_e60ca14a · 4 models · 2 measured signatures · 1 inferred tokenizerexact signature tk_06ffa14a · 3 models template variant tk_e60ca14a · 1 model

Llama2fg_5348f9f6 · 3 models · 2 measured signatures · 1 inferred tokenizerexact signature tk_5348f9f6 · 2 models template variant tk_03ecdf53 · 1 model

Nova · group Afg_76e424dd · 3 models · 1 measured signature · 1 inferred tokenizer

Mistral · group Bfg_ddc193e9 · 3 models · 1 measured signature · 1 inferred tokenizer

Llama4fg_20b9d005 · 3 models · 2 measured signatures · 1 inferred tokenizerexact signature tk_59e2520c · 2 models template variant tk_20b9d005 · 1 model

Moonshot AI · group Afg_ea6068bf · 3 models · 1 measured signature · 1 inferred tokenizer

Grok · group Afg_a17bbc1d · 3 models · 1 measured signature · 1 inferred tokenizer

Upstagefg_865d84ce · 2 models · 1 measured signature · 1 inferred tokenizer

Cohere · group Afg_2729e470 · 2 models · 1 measured signature · 1 inferred tokenizer

Gemmafg_ba03436b · 2 models · 1 measured signature · 1 inferred tokenizer

Claudefg_2d06485c · 2 models · 1 measured signature · 1 inferred tokenizer

Aion Labs · group Afg_d3c6ae80 · 2 models · 1 measured signature · 1 inferred tokenizer

When one family name appears on several groups (Qwen has 4; Mistral has 4; Tencent has 4; Nova has 3; Moonshot AI has 3; Gemini has 2; Grok has 2; Cohere has 2; Aion Labs has 2; Poolside has 2), those are distinct measured vocabularies from the same broader model family: tokenizer generations, lettered group A, B and onwards in size order. The letters are ours; the evidence does not name the generations.

One of a kind, so far

A unique EXACT fingerprint does not take a model out of its family: the nearest relative, and how close it comes, is shown on each row

A model lands here when its exact 50-number fingerprint has no twin yet, which usually means a vocabulary revision inside its own family (Qwen3.8 sits close to the Qwen line without matching it exactly) or relatives that are still unmeasured.

Reading the Atlas

A group gathers every model whose measured fingerprints are the same vocabulary: exact signature matches, plus signatures one uniform template shift apart. Group membership is lineage evidence, not release identification, and the methodology records exactly how much the fingerprints can distinguish. The Atlas's working face is the unknowns queue: what is still unplaced, and what measurement has already placed.