llm-rs-python

Adds the ability to extend the context length of models via the RoPE_scaling parameter.

Simplified the interaction with other GGML based repos. Like TheBloke/Llama-2-7B-GGML created by TheBloke.

Fixed many gpu acceleration bugs in rustformers\llm and improved performance to match native ggml.

Adds support for Metal/CUDA and OpenCL acceleration for LLama-based models.

Adds CI for the different acceleration backends to create prebuild binaries

• Added support for the haystack library
• Support "BigCode" like models (e.g. WizardCoder) via the gpt2 architecture

AutoModel compatible models will now use the official tokenizers library, which improves the decoding accuracy, especially for all non llama based models.

If you want to specify a tokenizer manually, it can be set via the tokenizer_path_or_repo_id parameter. If you want to use the default GGML tokenizer the huggingface support can be disabled via use_hf_tokenizer.

Added support for q5_0,q5_1 and q8_0 formats.

Added the stream method to each model, which returns a generator that can be consumed to generate a response.

⚠️ The GGML quantization format was updated again, old models will be incompatible ⚠️

AutoModel can now automatically download GGML converted models and normal Transformer models from the Huggingface Hub.

Added the ability to automatically convert any supported model from the Huggingface Hub via the AutoConverter.

Models which were converted this way, can be easily quantized or loaded via the AutoQuantizer or AutoModel without the need to specifiy the architecture.

The ability to quantize models is now available for every architecture via quantize.

• Added support for Mosaic ML's MPT models.
• Added support for LoRA adapters for all architectures.

⚠️Caution⚠️
Due to changes in the ggml format old quantized models are not supported anymore!

Added the tokenize and decode functions to each model, to enable access to the internal tokenizer.

The generation of tokens is now GIL free, meaning other background threads can run at the same time.

Since llama-rs was renamed to llm and now supports multiple model architectures, this wrapper was also expanded to support the new trait system and library structure.

Supported architectures for now:

• Llama
• GPT2
• GPTJ
• GPT-NeoX
• Bloom

The loader was also reworked and now supports the mmap-able ggjt. To support this the SessionConfig was expandend with the prefer_mmap field.