Token counting CLI and library for multiple LLM tokenizers. Count tokens and estimate API costs across OpenAI, Anthropic, Meta, and Mistral models from a single interface.
Every LLM provider uses different tokenization. Before sending a prompt, you need to know how many tokens it contains � for cost estimation, context window management, and prompt optimization. token-counter gives you accurate counts for 10+ models in one command.
pip install token-counterOr install from source:
git clone https://github.com/nripankadas07/token-counter.git
cd token-counter
pip install -e .Count tokens across all supported models:
token-counter count "Your prompt text goes here"Count for specific models:
token-counter count -m gpt-4o -m claude-3.5-sonnet "Hello, world!"Count tokens in a file:
token-counter count --file prompt.txtPipe from stdin:
cat prompt.txt | token-counter countList all supported models:
token-counter modelsfrom token_counter import TokenCounter, count_tokens, estimate_cost
# Quick one-liner
tokens = count_tokens("Hello, world!", model="gpt-4o")
print(f"Tokens: {tokens}")
# Detailed counting
counter = TokenCounter()
result = counter.count("Your prompt here", "claude-3.5-sonnet")
print(f"Tokens: {result.token_count}")
print(f"Input cost: ${result.input_cost:.6f}")
print(f"Output cost: ${result.output_cost:.6f}")
# Compare across models
results = counter.count_multi("Your prompt here", ["gpt-4o", "gpt-4", "claude-3.5-sonnet"])
for r in results:
print(f"{r.model}: {r.token_count} tokens (${r.input_cost:.6f})")
# Quick whitespace estimate (no tokenizer needed)
estimate = TokenCounter.estimate_whitespace("Hello world")| Model | Provider | Encoding | Input $/1K | Output $/1K |
|---|---|---|---|---|
| gpt-4o | OpenAI | o200k_base | $0.005 | $0.015 |
| gpt-4o-mini | OpenAI | o200k_base | $0.00015 | $0.0006 |
| gpt-4-turbo | OpenAI | cl100k_base | $0.01 | $0.03 |
| gpt-4 | OpenAI | cl100k_base | $0.03 | $0.06 |
| gpt-3.5-turbo | OpenAI | cl100k_base | $0.0005 | $0.0015 |
| claude-3.5-sonnet | Anthropic | cl100k_base* | $0.003 | $0.015 |
| claude-3-opus | Anthropic | cl100k_base* | $0.015 | $0.075 |
| claude-3-haiku | Anthropic | cl100k_base* | $0.00025 | $0.00125 |
| llama-3-70b | Meta | cl100k_base* | free | free |
| mistral-large | Mistral | cl100k_base* | $0.004 | $0.012 |
*Non-OpenAI models use cl100k_base as a close approximation. Actual token counts may differ by ~5%.
The main class for counting tokens.
count(text, model)� Count tokens for a single model. ReturnsTokenCount.count_multi(text, models=None)� Count across multiple models. Returns list ofTokenCount.estimate_whitespace(text)� Quick word-based estimate (~1.3x word count). Static method.
text_length: int� Character count of input texttoken_count: int� Number of tokensmodel: str� Model name usedencoding_name: str� Tiktoken encoding nameprovider: str� Model providerinput_cost: float | None� Estimated input cost in USDoutput_cost: float | None� Estimated output cost in USD
count_tokens(text, model="gpt-4o")� Returns token count asint.estimate_cost(text, model="gpt-4o", direction="input")� Returns cost asfloat.
token_counter/
��� main.py # Core engine: TokenCounter, MODEL_REGISTRY, cost estimation
��� cli.py # Click CLI with Rich table output
The design separates the tokenization engine from the CLI layer. TokenCounter caches tiktoken encodings for performance when counting across multiple inputs. The model registry is a simple dict that maps model names to their encoding and pricing, making it trivial to add new models.
MIT License � Copyright 2024 Nripanka Das