ttok

Count and truncate text based on tokens

APACHE-2.0 License

Downloads

1.3K

Stars

263

Committers

View Code on GitHub View on X

Ecosystems: Python

Bot releases are visible (Hide)

ttok - 0.3 Latest Release

Published by simonw 6 months ago

New --allow-special option for allowing special tokens: ttok '<|endoftext|>' --encode --allow-special #13

ttok - 0.2

Published by simonw over 1 year ago

--encode now encodes text to integer tokens and displays them. This has been renamed from --tokens.
--decode against a sequence of integer tokens now turns those back into text. #7
--tokens for either of those options outputs the raw tokens in a format that helps show if they have a leading space or similar. #4

ttok - 0.1

Published by simonw over 1 year ago

Initial release. echo input.txt | ttok to count tokens, use -t 100 to truncate to 100 tokens, or --tokens to see what those integer tokens would be. #1

Package Rankings

Top 11.99% on Pypi.org

Badges

Extracted from project README

PyPI

Changelog

Related Projects

minbpe

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

16 Feb 2024 9,074

token2index

A lightweight but powerful library to build token indices for NLP tasks, compatible with major De...

TTS

🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production

20 May 2020 34,030

chatgpt-prompt-wrapper

cli tool to wrap ChatGPT Python API

llm-gpt4all

Plugin for LLM adding support for the GPT4All collection of models

09 Jul 2023 177

openai-messages-token-helper

A utility library for dealing with token counting for messages sent to an LLM (currently OpenAI m...

datasette-openai

SQL functions for calling OpenAI APIs

Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

02 Jul 2021 1,323

starcoder2

Home of StarCoder2!

08 Dec 2023 1,732

TTS

Deep learning for Text to Speech (Discussion forum: https://discourse.mozilla.org/c/tts)

23 Jan 2018 8,880

tokenize-rt

A wrapper around the stdlib `tokenize` which roundtrips.