Skip to main content
Mobius
Intermediate

What is a token

Also known as: tokens, AI token

Definition

A token is a piece of a word that an AI language model reads and writes. Usage is counted and billed in tokens, and the same text takes more in some languages.

Why it matters

For a business that works in Hebrew or Russian, tokens explain why the same instructions fill more of the window and use more per question than in English.

Common mistakes

Assuming one token is one word, and underestimating usage, especially in Hebrew and Russian.

Related terms

Worked exampleIllustration

The same six rules in three languages, at a fictional pilates studio

A fictional neighbourhood pilates studio has six policy rules, written in English, Hebrew and Russian with the same meaning. The rules are invented, and the token counts were measured on 2026-09-25 with OpenAI's open source tiktoken library.

  1. The text

    The six rules as one paragraph in each language: cancelling up to 24 hours before, moving a class, lateness, adults only, a towel and water.

  2. Current tokenizer

    Counted with o200k_base, the tokenizer of OpenAI's current models.

    EN 63, HE 86, RU 82

  3. The gap

    Hebrew takes about 1.4 times the English count, Russian about 1.3.

    86 ÷ 63 ≈ 1.4, 82 ÷ 63 ≈ 1.3

  4. Older tokenizer

    Counted with the older cl100k_base, the gap was much wider: about 3.3 and 2.2 times.

    EN 63, HE 205, RU 136

  5. What it means

    The Hebrew sheet fills more of the context window, and on a plan billed by tokens it uses more per question.

Tokens are the meter. The language, and the tokenizer, change how fast it runs.

Illustration

The same rule, three token counts

English: one rule, 23 tokens

  • You
  • can
  • cancel
  • a
  • class
  • up
  • to
  • ␣
  • 24
  • hours
  • before
  • it
  • starts
  • .
  • A
  • later
  • cancellation
  • counts
  • as
  • a
  • used
  • class
  • .

The six rules as one paragraph, o200k_base

  • English63
  • Hebrew86
  • Russian82

The same six rules: 63 tokens in English, 86 in Hebrew, 82 in Russian. Hebrew takes about 1.4 times the English count.

Measured on 25 September 2026 with OpenAI’s open source tiktoken 0.12.0, tokenizers o200k_base and cl100k_base, no API call.

A tokenizer cuts text into the pieces a model counts. The same rule in three languages is cut into a different number of pieces.

Definition

A token is the unit a language model reads and writes: a short word, a piece of a longer word, a number, a space or a punctuation mark. A tokenizer splits text into tokens before the model sees it, and the model writes its answer one token at a time. Tokens measure two things: how much fits in the model's context window, and how much a request uses on a plan that charges by use. The same content takes a different number of tokens in different languages, and how many depends on the tokenizer.

Why it matters

In an example measured for this page, six policy lines took 63 tokens in English, 86 in Hebrew and 82 in Russian with a current tokenizer, and 205 and 136 with an older one. An old article about Hebrew and AI may quote a gap that newer tokenizers have narrowed.

Directly related: Context Window, LLM, Inference.

Improvement tips

  • Check a long document with a token counter before you paste it, especially in Hebrew or Russian.
  • Keep standing instructions short, because they are read, and counted, with every question.
  • Ask for the answer length you need, because the answer's tokens count too.
  • Watch usage in automated flows, where one customer message can trigger several model calls.

Common mistakes

  • Applying an English rule of thumb to Hebrew text: in the measured paragraph English averaged about 4.2 characters a token, Hebrew about 2.6.
  • Mixing up two limits: an input too long for the context window is not fully seen, while an answer that reaches the output limit is cut off.
  • Confusing an AI token with a crypto token or with an API access key.

Related terms

AI and DataIntermediate

Context Window

A context window is how much text, in tokens, a model can see at once: instructions, files, the chat so far and its reply. What falls outside, it cannot use.

AI and DataIntermediate

LLM

An LLM (large language model) is AI trained on masses of text to predict the next piece of a sentence. That is how chat tools write answers, right or wrong.

AI and DataIntermediate

Inference

Inference is the moment a trained AI model does its job: it takes a new question and produces an answer. It happens on every request, so it is the ongoing cost.

AI and DataBeginner

Prompt

A prompt is the instruction you give an AI tool: the question, the task, the context and the format you want. A clearer prompt gets a more useful answer.

AI and DataAdvanced

Embedding

An embedding is a list of numbers that stands for the meaning of a text or image, so texts that mean the same land close together, even with different words.

TechnologyIntermediate

API

An API (application programming interface) is the agreed way one program asks another for data or an action and gets an answer, so your tools can work together.

Quick check

The same paragraph takes 63 tokens in English and 86 in Hebrew with one tokenizer. What follows?

Choose an answer

Frequently asked questions

What is a token in simple words?
A small piece of text the model reads and writes: a short word, part of a longer word, a number or a punctuation mark. The model splits your text into tokens before reading it, and writes its answer one token at a time.
How many characters is a token?
It depends on the language and the tokenizer. In a paragraph measured for this page with a current tokenizer, English averaged about 4.2 characters a token, Russian about 3.8 and Hebrew about 2.6.
Does Hebrew take more tokens than English?
Usually, yes. Six rules measured for this page took 63 tokens in English and 86 in Hebrew with a current tokenizer, about 1.4 times. With an older tokenizer the Hebrew took 205, more than three times. Newer tokenizers narrowed the gap, they did not close it.
Why does the same text take different numbers of tokens in different languages?
Each tokenizer has a fixed set of pieces, and some languages are covered by longer pieces than others. A NeurIPS 2023 study found that the same text can take up to 15 times as many tokens in one language as in another, which affects cost, speed and how much fits in the window.
Why do tokens matter for my bill?
Many AI services measure use in tokens: what you send and what the model writes. Some price input and output tokens differently, as Anthropic's model overview shows. This page gives no prices: check your tool's own pricing page.
What happens if I send too many tokens?
There are two different limits. If the input is longer than the context window, the model cannot see all of it, and the app cuts it, summarises it or refuses. If the answer reaches the output limit, it stops in the middle. Splitting the task or sending only the needed passages helps with both.
Can I run out of tokens in a chatbot?
In two different ways. One chat can grow beyond the context window, and then a new chat with a short summary helps. Separately, your plan may cap use for a period, which is a question about the plan, not the model.
Is an AI token a crypto token?
No. In AI a token is a unit of text used to measure what a model reads and writes. It has no value of its own, and it is not an API access key either, which is also sometimes called a token.
Do I need to count tokens myself?
No. Tools count them automatically, and many show your usage in a dashboard. A token counter helps before you paste a long document, to see whether it fits.
How can I use fewer tokens without worse answers?
Keep standing instructions short, send only the passages the question needs, remove repeated text from documents, and ask for the answer length you actually need.
Is an old figure about Hebrew and tokens still right?
Maybe not. With an older tokenizer our Hebrew example took about 3.3 times the English count, with a current one about 1.4. When you read a figure about Hebrew and AI, check which tokenizer it measured.
Do answers count as tokens too?
Yes. The model writes its answer in tokens, and they count toward the context window and, on plans that charge by use, toward usage. Asking for a short answer when that is enough uses less.

Sources: Measured for this page with tiktoken 0.12.0 (github.com/openai/tiktoken), tokenizers o200k_base and cl100k_base, the six rules in English, Hebrew and Russian, 2026-09-25, Petrov, La Malfa, Torr and Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023 (arXiv 2305.15425), read 2026-09-25, Anthropic, "Models overview", Claude documentation, pricing listed per input and per output token (platform.claude.com/docs/en/about-claude/models/overview), read 2026-09-25

Last reviewed: 2026-09-25

I have worked with

What Clients Say

From different fields, at different stages of business

Dan Manto

Eclipse Capital

Real Estate Investment and Finance, USA

Arty McLabin

GameReady

Game Development Education and Outsourcing, International

Anna, Beautician

Anna

Beautician

After I finished a cosmetology course at a leading company, I was confused. I didn't know how to open a business, what to sell from home, or how to bring in clients. Since I reached out to Mobius, everything has changed. We built an organized plan with clear steps and no unnecessary risks. Today I see results, and I'm growing and developing every month.

Mark, Massage therapist, Gan Yavne

Mark

Massage therapist, Gan Yavne

When I finished a massage course, I imagined a different world. I found out it's a hard field, and financial stability isn't easy to find. When I came to Mobius, something changed. We started an organized process and built a business plan. Without Mobius? I probably would have given up. Today I believe in myself and in my own path.

Wondering where AI actually fits your business?

Alex helps Israeli businesses use AI and data where it pays off, and skip it where it does not. The first call is free.

+972 055-248-6151. Better on WhatsApp: I am in meetings most of the day and answer as soon as I am free.