Token counting is the process of calculating the exact number of tokens a specific input will consume before sending it to a large language model, allowing developers to predict costs, manage context window limits, and optimize application performance.
Sending a prompt to a large language model without counting the tokens first is a bit like throwing clothes into a suitcase without checking the airline's weight limit. You might get away with it for a while, but eventually, you are going to hit a hard limit or get hit with an unexpected bill. It turns out that knowing how many tokens your input will use is a distinct and surprisingly complex engineering challenge — one that goes well beyond counting words or characters.
When you build an AI application, you are operating within strict constraints. Every model has a maximum context window — the total number of tokens it can process at once. Every model also charges you based on the number of tokens you send and receive. If you do not know how many tokens your prompt contains, you cannot guarantee that your application will work, and you certainly cannot predict how much it will cost to run at scale.
This is why token counting has evolved from a simple string length estimation into a critical component of the modern AI software stack.
The Mechanics of Client-Side Estimation
The most common way developers handle this challenge is through client-side token counting. This involves running a local piece of software that mimics the AI model's tokenization process, allowing you to calculate the token count on your own servers before making a network request to the AI provider.
For developers working with OpenAI models, the standard tool for this job is an open-source library called tiktoken (OpenAI, 2024). This library is designed to be exceptionally fast, capable of processing gigabytes of text in seconds. When you pass a string of text into tiktoken, you must specify which encoding the target model uses. GPT-4, for example, uses an encoding called cl100k_base, while newer models use o200k_base.
The library then applies the exact same Byte Pair Encoding rules that the actual model uses, returning an array of integers. By simply counting the length of this array, you get a highly accurate estimate of how many tokens your text will consume.
However, client-side counting has significant limitations. While libraries like tiktoken are excellent at counting raw text, they often struggle to account for the hidden overhead introduced by the API itself. When you send a request to an AI provider, your text is wrapped in specific formatting tokens that tell the model where a user message ends and an assistant message begins. If you are using function calling or providing a system prompt, the provider injects additional tokens behind the scenes to structure that information for the model.
Because these formatting rules are often proprietary and subject to change, client-side token counters can sometimes underestimate the true token count of a complex API request. Each tool definition you include in a function-calling request, for instance, can add anywhere from 50 to 100 tokens of overhead — and that cost multiplies with every tool you register (tokenmix.ai, 2026).
The Shift to Server-Side Counting Endpoints
To solve the discrepancy between client-side estimates and actual API billing, major AI providers have begun introducing dedicated server-side token counting endpoints.
Instead of relying on a local library to guess how the provider will format your request, you send your exact, fully formatted payload to a specific API endpoint designed solely for counting. The provider processes the payload exactly as it would for a real generation request, but instead of generating a response, it simply returns the exact token count (OpenAI, 2024).
This approach eliminates the guesswork. It perfectly accounts for the hidden tokens used in message formatting, system prompts, and tool definitions. More importantly, it provides a reliable way to count tokens for inputs that local libraries cannot easily process, such as images and documents.
Anthropic offers a similar feature for its Claude models, allowing developers to proactively manage rate limits and make smart model routing decisions without incurring billing charges for the counting process itself (Anthropic, 2024). By using these endpoints, developers can build robust validation checks into their applications, ensuring that a prompt will never exceed the context window before committing to the cost and latency of a full generation request.
The Multimodal Mathematics of Images and Audio
Text is relatively straightforward to count, but modern AI models are multimodal, meaning they can process images, audio, and video alongside text. Counting tokens for these formats requires an entirely different set of mathematical rules.
You cannot simply pass an image through a text tokenizer. Instead, AI providers use specific formulas based on the dimensions and resolution of the media. When sending an image to Google's Gemini models, if both dimensions of the image are under 384 pixels, it is counted as a flat 258 tokens. If the image is larger, it is cropped and scaled into tiles of 768 by 768 pixels, with each tile consuming 258 tokens (Google, 2024).
Audio and video are typically counted based on duration rather than resolution. Gemini converts video to tokens at a fixed rate of 263 tokens per second, while audio is processed at 32 tokens per second.
This means that a seemingly simple prompt containing a high-resolution image or a short video clip can consume thousands of tokens before a single word of text is even considered. Developers building multimodal applications must implement logic to calculate these media token costs dynamically, often resizing or compressing images on the client side to fit within a specific token budget before sending them to the API.
The Hidden Overhead of Structured Data
One of the most common ways developers interact with AI models is by passing structured data, such as JSON, into the prompt. You might pull a user's profile, their recent purchase history, and a catalog of products from your database, format it all as a JSON object, and ask the model to make a recommendation.
While this approach is highly effective for providing context, it is notoriously inefficient from a token counting perspective. JSON is designed to be human-readable, which means it is full of repetitive structural characters — curly braces, quotation marks, and repeated field names that appear on every single record.
When you pass a large JSON object to an AI model, you are paying for every one of those structural characters. A nested JSON structure that takes up 4,000 tokens might only contain 1,000 tokens of actual, useful information. The rest is formatting overhead.
This inefficiency becomes a massive problem when scaling applications. If you are processing millions of records, that structural overhead translates directly into wasted money and exhausted context windows. To combat this, developers are increasingly moving away from verbose JSON in favor of more token-efficient serialization formats. Flattening nested hierarchies, removing redundant keys, and switching to simpler formats like CSV or custom delimited strings can reduce the token footprint of structured data by up to 70 percent (Patel, 2025).
For teams building complex applications that require passing large amounts of structured data into AI workflows, platforms like Sandgarden can help manage this kind of infrastructure complexity. Sandgarden is a modularized platform for prototyping, iterating, and deploying AI applications, removing the overhead of manually wrestling with serialization inefficiencies so you can focus on what the application actually does.
It is also worth noting that the problem is not limited to JSON. XML is even more verbose, and Markdown — while far more compact — still introduces structural tokens through its heading markers, asterisks, and code fences. Any format that adds characters to convey structure rather than content is adding tokens to your bill. The practical takeaway is that before you finalize how your application formats data for the model, you should run it through a token counter and compare the count against a stripped-down alternative. The difference is often shocking.
Managing Budgets in Agentic Workflows
Token counting becomes exponentially more difficult when you move from simple, single-turn prompts to complex, multi-agent workflows.
In an agentic system, an AI model is given a goal and a set of tools, and it iteratively decides which tools to use, observes the results, and plans its next steps. Every time the agent takes an action, the results of that action must be appended to the conversation history and sent back to the model for the next step.
This creates a compounding token consumption problem. If an agent takes ten steps to solve a problem, the context window grows larger with every single step. The system prompt and the tool definitions are re-sent every time. The results of step one are sent in step two, step three, and all the way through step ten.
Recent academic studies analyzing execution traces of multi-agent software development frameworks have found that this iterative process — particularly during stages like automated code review — accounts for the vast majority of token consumption in agentic systems (Salim et al., 2026). The initial generation of code is relatively cheap; it is the continuous, compounding loop of refinement and verification that drains the token budget.
To build sustainable agentic applications, developers must implement aggressive token counting and context management strategies. This often involves dynamically summarizing older parts of the conversation, dropping the results of failed tool calls, or using tiered architectures where a smaller, cheaper model handles the iterative routing while a larger model is only invoked for the final generation.
Some teams implement what is known as a sliding context window, where only the most recent N turns of the conversation are kept in full, while everything older is compressed into a rolling summary. Others use token budgets as a first-class architectural constraint, building hard limits into their agent orchestration logic so that no single workflow can exceed a predetermined cost ceiling. Without this kind of discipline, a single runaway agentic task can consume millions of tokens — and the bill for that will arrive whether the task succeeded or not.
The Impact of Reasoning and Thinking Tokens
The introduction of advanced reasoning models has added a fascinating new dimension to token counting. Models designed to "think" before they respond generate a hidden chain of thought used to solve complex logic or math problems.
While you do not always see these thinking tokens in the final output provided to the user, they still occupy space in the model's context window, and crucially, they are billed as output tokens. Because these models can generate thousands of thinking tokens for a single complex query, the cost and context usage can skyrocket unexpectedly.
To manage this, providers allow developers to set a specific budget for thinking tokens. When using Anthropic's extended thinking features, for example, you can define exactly how many tokens the model is allowed to spend on its internal reasoning process (Anthropic, 2024).
This requires a delicate balancing act. Set the thinking token budget too low, and the model might not have enough space to solve the problem, resulting in a degraded or incomplete answer. Set it too high, and you risk paying for unnecessary computation. Researchers are actively exploring token-budget-aware reasoning frameworks that can dynamically estimate and adjust the required number of reasoning tokens based on the complexity of the specific problem being asked (Han et al., 2024).
Token Counting in Training and Fine-Tuning
While most discussions about token counting focus on inference — the process of sending a prompt and getting a response — it is equally critical during the model training and fine-tuning phases.
When you prepare a custom dataset to fine-tune an open-source model or customize a commercial API, you must carefully count the tokens in your training examples. Fine-tuning costs are calculated based on the total number of tokens in the training file multiplied by the number of epochs — the number of times the model passes through the data.
If your training data is full of verbose formatting, redundant system prompts, or unnecessary boilerplate text, you will pay to train the model on that noise over and over again. Efficient token counting and data preparation before fine-tuning ensures that your training budget is spent entirely on the high-value, domain-specific knowledge you actually want the model to learn.
Furthermore, if you can successfully fine-tune a model to understand your specific domain, you can often drastically reduce the size of the system prompt required during inference. By moving the instructions from the prompt into the model's weights, you permanently lower the token count of every future API call.
This is a powerful but underappreciated strategy. A system prompt that runs to 2,000 tokens and is sent with every single API request can be partially or entirely eliminated after a successful fine-tune. At scale — say, a million requests per day — that is two billion tokens of savings daily. The fine-tuning investment pays for itself quickly when viewed through this lens.
The same logic applies to few-shot examples, which are sample input-output pairs included in the prompt to teach the model how to respond. Including three or four detailed examples in every prompt is a common technique for improving model accuracy, but it is also expensive. A well-fine-tuned model can often match or exceed the accuracy of a few-shot prompted model while using a fraction of the tokens per request.
Putting It All Together
The table below summarizes four of the most impactful token optimization strategies, along with their typical reduction ranges and the scenarios where they deliver the most value.
Counting tokens is no longer just a neat trick for estimating a monthly bill. It is a fundamental engineering practice that dictates the architecture, scalability, and economic viability of modern AI applications. Whether you are stripping the curly braces out of a JSON payload, calculating the exact tile count of a high-resolution image, or setting a strict budget for a reasoning model's internal monologue, mastering the mathematics of token counting is the only way to ensure your AI systems remain both powerful and predictable.
