Skip to main content
Mobius
Intermediate

What is inference

Also known as: model inference, running a model

Definition

Inference is the moment a trained AI model does its job: it takes a new question and produces an answer. It happens on every request, so it is the ongoing cost.

Why it matters

Every question your customers or your team ask an AI tool is one inference run.

Common mistakes

Budgeting only for the setup and forgetting that inference is paid again with every question, so the bill grows with use.

Related terms

Worked exampleIllustration

A month of questions at a fictional pilates studio

A fictional neighbourhood pilates studio answers customer questions on WhatsApp with an AI assistant. The numbers are approximate and invented for this example.

  1. One question

    One customer question is one inference run: the model reads, then writes an answer.

  2. One run

    Each run reads the question (about 30 words) and the studio's policy sheet (about 1,200 words), and writes an answer (about 80 words).

    30 + 1,200 + 80 = 1,310 words

  3. A month

    The studio gets 300 questions a month.

    1,310 × 300 = 393,000 words

  4. A longer sheet

    The owner doubles the policy sheet to 2,400 words. The same 300 questions now read 2,510 words each.

    2,510 × 300 = 753,000 words

Almost double the usage without one extra customer. The vendor paid for training once. You pay for inference every time someone asks, so keep what you send with each question short.

Illustration

Every question is a run

One run: 1,310 words

  • Question, about 30 words
  • Policy sheet, 1,200 words
  • Answer, about 80 words

A month: 300 questions × 1,310 words = 393,000

White line: today’s 393,000 words a month

300 questions × 1,310 words = 393,000 words a month, most of it the sheet read again with every question. Double the sheet to 2,400 words and the same questions read 753,000.

Each customer question is one inference run: the model reads the question and the policy sheet, then writes an answer. The month adds up run by run.

Trained once, used every time

Training happens once, by the vendor, before anyone asks a question. Inference runs on every question, every month, for as long as the model is used.⁨Training⁩⁨once, by⁩⁨the vendor⁩⁨Model⁩⁨Inference⁩⁨every question,⁩⁨every month⁩
Training happens once, by the vendor, before anyone asks a question. Inference runs on every question, every month, for as long as the model is used.

Definition

Inference is the stage where a model that has already been trained is used: it receives a new input, such as a customer's question, and applies what it learned to produce an output, such as an answer, a label or a forecast. The model does not change while it answers. Every question sent to a chat tool or an AI assistant is one inference run, and each run is measured in tokens: the question, any document sent with it and the answer all count. Inference can run on the vendor's servers or, with a small open model, on your own computer, which keeps the data in the office and is usually slower.

Why it matters

The vendor paid for training once, inference happens every time, so usage drives the bill. The longer the text you send with each question, the more each run uses, even when the number of questions stays the same.

Directly related: Model Training, AI Model, Token.

Improvement tips

  • Do not send the same question twice when a saved answer will do: keep ready replies for the questions customers ask every day.
  • Send only the part of a document the question needs, not the whole file, because everything you send is read again on every run.
  • Check how your tool charges for use: per request, per amount of text, or as limits inside a subscription.

Common mistakes

  • Confusing training, where a model learns from examples, with inference, where the finished model answers new questions.
  • Attaching a long document to every question when only one paragraph of it is needed.
Training vs Inference

Related terms

AI and DataIntermediate

Model Training

Model training is how an AI model learns: it sees many examples, checks its guesses against the right answers and adjusts itself until the errors shrink.

AI and DataIntermediate

AI Model

An AI model is the trained file at the heart of an AI tool: it takes an input, applies patterns learned from data, and returns a prediction, a label or text.

AI and DataIntermediate

Token

A token is a piece of a word that an AI language model reads and writes. Usage is counted and billed in tokens, and the same text takes more in some languages.

AI and DataIntermediate

Context Window

A context window is how much text, in tokens, a model can see at once: instructions, files, the chat so far and its reply. What falls outside, it cannot use.

AI and DataIntermediate

LLM

An LLM (large language model) is AI trained on masses of text to predict the next piece of a sentence. That is how chat tools write answers, right or wrong.

AI and DataIntermediate

Machine Learning

Machine learning is the part of AI where software learns rules from examples instead of being told them, then uses those rules to predict or sort new cases.

Quick check

A studio doubles the policy sheet its assistant reads with every question. What happens to inference usage for the same 300 questions?

Choose an answer

Frequently asked questions

What is inference in AI, in simple words?
It is the moment a trained model does the work: it gets a new question and produces an answer. The learning happened earlier, in training. Inference is using what was learned, and it happens again with every question.
Why is it called inference?
In ordinary English, to infer is to reach a conclusion from what you know. A model does something similar in form: from the patterns it learned, it produces an output for an input it has not seen. The word is a technical label, not a claim that the model reasons like a person.
Is inference the same as training?
No. Training is when the model learns from a large set of examples, usually once and by the vendor. Inference is every later use of the finished model. Training is paid once, inference every time the model is used.
Does the model learn from the questions it answers?
Not while it answers: during inference the model applies what it already learned and stays the same. Whether your conversations may later be used to train a future version is a setting of your plan, so check it.
How do I pay for inference?
By use. Depending on the tool, you pay per request, per amount of text measured in tokens, or within limits included in a subscription. This page gives no prices, because they change often: check your tool's own pricing page.
Why does usage grow when the number of customers stays the same?
Because each run reads everything sent with the question. A longer policy sheet, an attached file or a long chat history is read again on every run. In the example above, doubling the sheet almost doubled the monthly usage.
How can I keep inference usage down?
Send the part of a document the question needs instead of the whole file, keep saved answers for questions that repeat, and ask for short answers where short is enough. Each of these cuts the text read on every run.
Can inference run on my own computer?
Yes, with a small open model and suitable software. The data then stays in the office, and answers are usually slower than from a large model on a vendor's servers. For most small businesses the vendor's tool is the practical choice.
Why is an answer sometimes slow?
The model writes the answer one piece at a time, so a long answer takes longer, and a long input takes longer to read. A busy period on the vendor's side can add delay. A short question with only the context it needs usually comes back faster.
Is inference only about chat tools?
No. Any trained model runs inference when it is used: a spam filter judging one email, till software suggesting how many loaves to bake, a phone recognising a face. Chat tools are simply the case most owners notice.
Do I need a technician to manage inference?
Not for a ready made tool: the vendor runs the model and you use the app. If you connect a model to your own website or systems through its API, someone technical sets it up and keeps an eye on usage.
What happens when many customers write at once?
Each message is its own run, so usage rises with every one. Vendors also limit how many requests an account may send in a period, and a busy moment can reach that limit. Check your tool's limits before a campaign that may bring many messages at once.

Sources: Google, Machine Learning Glossary, entry "inference" (developers.google.com/machine-learning/glossary), read 2026-09-25, Anthropic, "Rate limits", Claude API documentation (platform.claude.com/docs/en/api/rate-limits), read 2026-09-25

Last reviewed: 2026-09-25

I have worked with

What Clients Say

From different fields, at different stages of business

Dan Manto

Eclipse Capital

Real Estate Investment and Finance, USA

Arty McLabin

GameReady

Game Development Education and Outsourcing, International

Anna, Beautician

Anna

Beautician

After I finished a cosmetology course at a leading company, I was confused. I didn't know how to open a business, what to sell from home, or how to bring in clients. Since I reached out to Mobius, everything has changed. We built an organized plan with clear steps and no unnecessary risks. Today I see results, and I'm growing and developing every month.

Mark, Massage therapist, Gan Yavne

Mark

Massage therapist, Gan Yavne

When I finished a massage course, I imagined a different world. I found out it's a hard field, and financial stability isn't easy to find. When I came to Mobius, something changed. We started an organized process and built a business plan. Without Mobius? I probably would have given up. Today I believe in myself and in my own path.

Wondering where AI actually fits your business?

Alex helps Israeli businesses use AI and data where it pays off, and skip it where it does not. The first call is free.

+972 055-248-6151. Better on WhatsApp: I am in meetings most of the day and answer as soon as I am free.