llm-inference-benchmark

docs

Access 14k+ open source AI models across 30+ tasks with the Bytez inference API ✨

transformer-deploy

Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer ...

LLM-As-Chatbot

LLM as a Chatbot Service

llm-infer-bench

GPTQ-for-LLaMa

4 bits quantization of LLaMA using GPTQ

quantized-training

Explore training for quantized models

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. D...

gpt4local

Openai-style, fast & lightweight local language model inference w/ documents