Token Counter for GPT-4, ChatGPT & tiktoken

What Is a Token Counter?

A token counter measures how many tokens a piece of text breaks into when an AI language model reads it. Tokens are the small chunks of text — whole words, parts of words, or punctuation marks — that models such as GPT-4 and ChatGPT process instead of raw letters. This tool uses tiktoken, the open-source tokenizer library published by OpenAI, so the counts match what OpenAI's own models and API would produce.

How Tokenization Works

A tokenizer does not simply count words. It runs text through an algorithm called byte pair encoding (BPE). The algorithm starts with individual bytes of text and repeatedly merges the most common adjacent pairs, based on patterns learned from a large body of text during training. The result is a fixed vocabulary of tens of thousands of common chunks: whole short words, frequent word fragments, and single characters for anything rare.

Because of this, one English word often becomes one token, but longer or unusual words can split into two or more. Spaces are usually attached to the start of the next token rather than counted on their own, and punctuation marks are typically their own tokens.

Encodings This Tool Supports

Tiktoken offers several encodings, each tied to a family of OpenAI models. This tool supports three:

cl100k_base is selected by default, since it covers the model family most people use today. The same text can produce a different token count under each encoding, because each one was trained on different text and merges characters differently.

How to Calculate a Token Count

There is no simple arithmetic formula for turning character count or word count into a token count, because BPE merges depend on the specific text and the trained vocabulary. The only exact method is to run the text through the tokenizer itself:

1. Pick the encoding that matches the target model.
2. Feed the text into that encoding's tokenizer.
3. Count the resulting list of token IDs. The length of that list is the token count.

As a rough estimate for English text, one token is close to four characters, or about three-quarters of a word. This is only a guide. Numbers, code, non-English text, and unusual spelling can shift the ratio a lot.

Worked Example

Take the sentence:

"Tokenization splits text into pieces."

Under the cl100k_base encoding, tiktoken breaks this into 7 tokens:

Token, ization, splits, text, into, pieces, .

Notice that "Tokenization" itself splits into two tokens, "Token" and "ization", while every other word stays whole. The final period is its own token. This is why a 38-character sentence with 6 words produces 7 tokens rather than 6.

Why Token Counts Matter

AI providers charge for API usage by the token, counting both the text sent in and the text generated in response, so a token count predicts cost before a request is sent. Every model also has a maximum context length, a limit on how many tokens a single request and its reply can contain together. That limit varies by model and changes as providers release new versions, so it is best checked in the current documentation for the model in use rather than assumed from memory.

Frequently Asked Questions

What is a token in AI language models? A token is a chunk of text a language model reads as one unit. It might be a whole short word, part of a longer word, a punctuation mark, or a space attached to the following word.

Why do different encodings give different token counts for the same text? Each encoding was trained on its own sample of text and built its own vocabulary of common chunks. A word that is common enough to be a single token in one vocabulary might not be in another, so it gets split differently.

Does punctuation count as a token? Yes. Punctuation marks are usually counted as their own tokens, or occasionally merged with the character next to them, depending on how often that combination appeared during the tokenizer's training.

Is a token the same as a word? No. Short, common words are usually one token each. Longer or less common words, and most words outside English, often split into two or more tokens.

How accurate is this tool? It counts tokens using OpenAI's own tiktoken library, running the same encoding logic OpenAI uses internally, so the count matches what an OpenAI API call would report for the same text and encoding.

Can this tool count tokens for non-OpenAI models? It uses tiktoken's encodings, which are built for OpenAI's models. Other providers, such as Anthropic or Google, use their own tokenizers, which can produce different counts for the same text.

References

1. OpenAI. "tiktoken." GitHub, https://github.com/openai/tiktoken.
2. Sennrich, Rico, et al. "Neural Machine Translation of Rare Words with Subword Units." arXiv:1508.07909, 2016, http://arxiv.org/abs/1508.07909.
3. Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv:2005.14165, 2020, http://arxiv.org/abs/2005.14165.