llama-tokenizer-js

TypeScript fix

• Add TypeScript types definition file
• Refactor tokenizer into a Class
• Allow passing custom vocab and merge data to tokenizer
• Allow passing custom tests to tokenizer test runner

• Fix bug in a function that was unused (so not affecting tokenizer results)
• Support very large inputs (previous version was not guaranteed to produce correct results for inputs larger than 100 000 characters, although in practice it would almost always produce correct results for large inputs)

Bugfix to support Next.js and other environments where performance.now() is not available.

Bugfix affecting results in extremely rare cases: equal prio merges are now always performed left-to-right.

Add support for different runtimes