LLM Token Count Estimator

A token is the basic unit a model reads and generates. Tokens are produced by a byte-pair encoding (BPE) or similar sub-word tokenizer, which learns the most frequent character sequences in the training data and stores them as a shared vocabulary. A single token can be a full word, a common prefix or suffix, part of a rare word, a single emoji, or a punctuation mark. For English prose, one token averages roughly four characters or about three-quarters of a word. Code, URLs, JSON, and non-Latin scripts tend to produce more tokens per character because their character sequences are less common in the tokenizer's vocabulary.

Each model family is trained with its own tokenizer and vocabulary. OpenAI's GPT-3.5 and GPT-4 use the cl100k_base encoding, while GPT-4o and the o1 series use the newer o200k_base encoding. Anthropic's Claude models use a proprietary Anthropic tokenizer, and Google's Gemini models use a SentencePiece tokenizer. Because the vocabularies differ, the same sentence can encode to different token counts on different models, typically within ten to twenty percent of each other for English prose but more divergent for code or non-English text.

The context window is the maximum number of tokens a model can read and generate within a single request. It includes the system prompt, the user prompt, the full conversation history, and the response. When you exceed the window, older context is truncated, which can silently drop instructions or facts the model needed. A large window gives room for long documents and long chats, but latency and cost grow with the number of tokens processed, so even with a two-million-token window it is usually cheaper and faster to keep prompts tight.

Most providers price input and output tokens separately and quote the rate per million tokens. Input tokens are everything you send to the model, including system prompts and conversation history. Output tokens are everything the model generates. Output is almost always more expensive than input because generation is compute-bound. A few providers also discount cached or reused input tokens. To estimate total cost for a call, multiply your input tokens by the input rate and your expected output tokens by the output rate, divide each by one million, then add the two figures together.

Producing an exact BPE token count requires shipping the full tokenizer vocabulary to the browser, which can be several megabytes of weights per encoding. This tool uses a characters-per-token heuristic calibrated for each model family, which gives a count within a few percent of the true tiktoken or SentencePiece number for typical English prose and is accurate enough for cost estimation and context-window planning. If you need the exact count for billing reconciliation, run the provider's official tokenizer against your final prompt before you send it.