Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

MIT License

Downloads
34K
Stars
154
Committers
14
View Code on GitHub Visit Website View on X
Ecosystems: VS Code Extension, Playwright, Windows, TypeScript, Windows UI Library (WinUI)

Commit Statistics

Past Year

All Time

Total Commits
36
73
Total Committers
10
15
Avg. Commits Per Committer
3.6
4.87
Bot Commits
0
2

Issue Statistics

Past Year

All Time

Total Pull Requests
22
38
Merged Pull Requests
16
32
Total Issues
8
11
Time to Close Issues
4 days
9 days
Package Rankings
Top 6.75% on Proxy.golang.org
Top 15.1% on Npmjs.org
Related Projects
llguidance

Low-level Guidance Parser

25 Jul 2024 17
tensorflow-directml-plugin

DirectML PluggableDevice plugin for TensorFlow 2

24 Sep 2021 185
Windows-Machine-Learning

Samples and Tools for Windows ML.

01 Mar 2018 1,015
dev-tunnels

Dev Tunnels SDK

04 Feb 2022 291
aici

AICI: Prompts as (Wasm) Programs

26 Sep 2023 1,916
PhoneticMatching

A phonetic matching library. Includes text utilities to do string comparisons on phonemes (the so...

21 Jun 2018 154
federal-ai-services

Secure AI Solutions

15 Aug 2024 3
typescript-error-deltas

Test popular repos on new versions of TypeScript: display new compiler errors

25 Jan 2022 22
mttl

Building modular LMs with parameter-efficient fine-tuning.

11 Jul 2022 76
MMdnn

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g...

16 Aug 2017 5,792
vscode-languagedetection

An npm package that uses ML to detect source code languages

13 Jul 2021 115
BioGPT

15 Aug 2022 4,292
fluentui-token-pipeline

The Fluent UI token pipeline generates source code for Fluent UI libraries.

20 Apr 2020 41
teams-ai

SDK focused on building AI based applications and extensions for Microsoft Teams and other Bot Fr...

06 Feb 2023 408
toktrie

LLM token utility library

05 Jul 2024 3