candle-vllm

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

MIT License

Stars

204

Ecosystems: Rust

Statistics for this project are still being loaded, please check back later.

Badges

Extracted from project README

An ecosystem of Rust libraries for working with large language models

Blazingly fast LLM inference.

Minimalist ML framework for Rust