Token Counter for GPT-4o, GPT-4 & tiktoken

What Is a Token Counter?

A token counter measures how many tokens a piece of text breaks into when an AI language model reads it. Tokens are the small chunks of text, such as whole words, parts of words, or punctuation marks, that models such as GPT-4 process instead of raw letters. This tool uses tiktoken, the open-source BPE tokenizer published by OpenAI for its models, so the counts match what OpenAI models that use the same encoding would produce.

How Tokenization Works

A tokenizer does not simply count words. It runs text through an algorithm called byte pair encoding (BPE). The algorithm starts with individual bytes of text and repeatedly merges the most common adjacent pair into a new token, based on patterns learned from a large body of text during training. The result is a fixed vocabulary of tens of thousands of common chunks: whole short words, frequent word fragments, and single characters for anything rare.

Because of this, one English word often becomes one token, but longer or unusual words can split into two or more. Spaces are usually attached to the start of the next token rather than counted on their own, and punctuation marks are typically their own tokens.

Encodings This Tool Supports

Tiktoken offers several encodings, each tied to a family of OpenAI models. This tool supports four:

Tiktoken's model table maps the models above to these encodings. GPT-4o, GPT-4.1, GPT-5 and the o-series models use o200k_base. GPT-4 and GPT-3.5 Turbo use cl100k_base. The tool lists the matching models next to each option. cl100k_base is selected by default.

The encoding definitions give r50k_base 50,257 tokens and p50k_base 50,281. The same text can produce a different token count under each encoding, because each one was trained on different text and merges characters differently.

How to Calculate a Token Count

There is no simple arithmetic formula for turning character count or word count into a token count, because BPE merges depend on the specific text and the trained vocabulary. The only exact method is to run the text through the tokenizer itself:

1. Pick the encoding that matches the target model.
2. Feed the text into that encoding's tokenizer.
3. Count the resulting list of token IDs. The length of that list is the token count.

As a rough estimate, tiktoken's documentation says each token corresponds to about 4 bytes of text on average. For plain English, one byte is one character, so that is about four characters per token. This is only a guide. Numbers, code, non-English text, and unusual spelling can shift the ratio a lot.

Worked Example

Take the sentence:

"Tokenization splits text into pieces."

Under the cl100k_base encoding, tiktoken breaks this into 7 tokens:

Token, ization, splits, text, into, pieces, .

Notice that "Tokenization" itself splits into two tokens, "Token" and "ization", while every other word stays whole. The final period is its own token. This is why a 37-character sentence with 5 words produces 7 tokens rather than 5. In this example, o200k_base, p50k_base and r50k_base give the same 7 tokens.

The encodings differ more on text outside English. The OpenAI Cookbook compares them on the Japanese phrase "お誕生日おめでとう" ("happy birthday"). It counts 14 tokens under r50k_base and p50k_base, 9 under cl100k_base, and 8 under o200k_base.

Why Token Counts Matter

AI providers charge for API usage by the token, counting both the text sent in and the text generated in response, so a token count predicts cost before a request is sent. Every model also has a maximum context length, a limit on how many tokens a single request and its reply can contain together. That limit varies by model and changes as providers release new versions, so it is best checked in the current documentation for the model in use rather than assumed from memory.

Frequently Asked Questions

What is a token in AI language models? A token is a chunk of text a language model reads as one unit. It might be a whole short word, part of a longer word, a punctuation mark, or a space attached to the following word.

Why do different encodings give different token counts for the same text? Each encoding was trained on its own sample of text and built its own vocabulary of common chunks. A word that is common enough to be a single token in one vocabulary might not be in another, so it gets split differently.

Does punctuation count as a token? Yes. Punctuation marks are usually counted as their own tokens, or occasionally merged with the character next to them, depending on how often that combination appeared during the tokenizer's training.

Is a token the same as a word? No. Short, common words are usually one token each. Longer or less common words, and most words outside English, often split into two or more tokens.

How accurate is this tool? It counts tokens using OpenAI's own tiktoken library, so the count is exact for the chosen encoding. It matches an OpenAI model only if that model uses the same encoding.

Which encoding should be selected? The one used by the target model. For GPT-4o, GPT-4.1, GPT-5 and the o-series models, that is o200k_base. For GPT-4 and GPT-3.5 Turbo, it is cl100k_base.

Can this tool count tokens for non-OpenAI models? It uses tiktoken's encodings, which are built for OpenAI's models. Other providers, such as Anthropic or Google, use their own tokenizers, which can produce different counts for the same text.

References

1. OpenAI. tiktoken (README). tiktoken on GitHub — a BPE tokeniser for use with OpenAI's models; on average each token corresponds to about 4 bytes.
2. OpenAI. tiktoken/_educational.py. tiktoken educational BPE code — BPE training starts from single bytes and repeatedly turns the most common pair into a new token.
3. OpenAI. tiktoken/model.py. tiktoken model-to-encoding table — which OpenAI models use cl100k_base, p50k_base, r50k_base and o200k_base.
4. OpenAI. tiktoken_ext/openai_public.py. tiktoken encoding definitions — vocabulary sizes of r50k_base (50,257) and p50k_base (50,281).
5. OpenAI. How to count tokens with tiktoken, OpenAI Cookbook. OpenAI Cookbook notebook — compare_encodings("お誕生日おめでとう") gives 14 tokens for r50k_base and p50k_base, 9 for cl100k_base and 8 for o200k_base.