gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.

MIT License

Downloads
435K
Stars
440
Committers
12
View Code on GitHub Visit Website View on X
Ecosystems: TypeScript
No releases found yet, please check back later.
Package Rankings
Top 2.41% on Npmjs.org
Related Projects
tokenx

📐 GPT token estimation and context size utilities without a full tokenizer

27 Nov 2023 13
compare-tokenizers

A test suite comparing Node.js BPE tokenizers for use with AI models.

20 Feb 2023 29
msgpack-javascript

@msgpack/msgpack - MessagePack for JavaScript / msgpack.org[JavaScript/TypeScript/ECMA-262]

12 Nov 2011 1,281
codepoint-iterator

Fast uint8array to utf-8 codepoint iterator for streams and array buffers by @okikio & @jonathant...

10 Jan 2023 3