tokenx

📐 GPT token estimation and context size utilities without a full tokenizer

MIT License

Downloads

42

Stars

13

View Code on GitHub

Ecosystems: TypeScript

Bot releases are visible (Hide)

tokenx - v0.4.0 Latest Release

Published by github-actions[bot] 3 months ago

🚨 Breaking Changes

Rename package to tokenx - by @johannschopplich (4a8ed)

View changes on GitHub

tokenx - v0.3.6

Published by github-actions[bot] 5 months ago

🚀 Features

Support GPT-4o "omni" model - by @johannschopplich (5e03f)

View changes on GitHub

tokenx - v0.3.5

Published by github-actions[bot] 6 months ago

🚀 Features

Update latest GPT model names - by @johannschopplich (2f6ff)

View changes on GitHub

tokenx - v0.3.4

Published by github-actions[bot] 9 months ago

🐞 Bug Fixes

Add missing gpt-4-turbo-preview - by @johannschopplich (a57b8)

View changes on GitHub

tokenx - v0.3.3

Published by github-actions[bot] 9 months ago

🚀 Features

Add latest GPT models like gpt-4-vision-preview - by @johannschopplich (7464c)

View changes on GitHub

tokenx - v0.3.2

Published by github-actions[bot] 11 months ago

No significant changes

View changes on GitHub

tokenx - v0.3.1

Published by github-actions[bot] 11 months ago

🚀 Features

Numeric sequences are often a single token - by @johannschopplich (ffa6a)

View changes on GitHub

tokenx - v0.3.0

Published by github-actions[bot] 11 months ago

🚀 Features

Support CJK languages - by @johannschopplich (11ec1)

View changes on GitHub

tokenx - v0.2.0

Published by github-actions[bot] 11 months ago

🚀 Features

Closer token estimations to GPT tokenizer - by @johannschopplich (a9cff)

View changes on GitHub

tokenx - v0.1.0

Published by github-actions[bot] 11 months ago

🚀 Features

Support accented characters - by @johannschopplich (c4133)

View changes on GitHub

Package Rankings

Top 34.44% on Npmjs.org

Related Projects

ax

The unofficial DSPy framework. Build LLM powered Agents and "Agentic workflows" based on the Stan...

23 Feb 2023 1,058

openai-chat-tokens

💬 Estimate the number of tokens an OpenAI chat completion will use

compare-tokenizers

A test suite comparing Node.js BPE tokenizers for use with AI models.

promptrix

Promptrix is a prompt layout engine for Large Language Models.