MInference
To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
MIT License
Bot releases are hidden (Show)
No releases found yet, please check back later.
Package Rankings
Top 35.48% on Pypi.org