tokeniser-api

A high-performance axum API for serving Hugging Face's Tokenizers.

Usage

If your tokeniser is not listed below, please open an issue or a pull request.

Meta LLama 3

To encode, simply cURL the endpoint like in the following.

curl -O 'https://winstxnhdw-tokeniser-api.hf.space/api/v1/llama3/encode' \
     -H 'Content-Type: application/json' \
     -d '{ "text": "Hello, world!" }'

To decode, simply cURL the endpoint like in the following.

curl -O 'https://winstxnhdw-tokeniser-api.hf.space/api/v1/llama3/decode' \
     -H 'Content-Type: application/json' \
     -d '{ "tokens": [ 0 ] }'