LocalAI

The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.

MIT License

Stars
20.5K
Committers
107
View Code on GitHub Visit Website View on X
Ecosystems: Kubernetes, Llama

Bot releases are hidden (Show)

LocalAI - v2.17.1 Latest Release

Published by mudler 4 months ago

Highlights

This is a patch release to address issues with Linux single binary releases. It also adds support for Stable diffusion 3!

Stable diffusion 3

You can use Stable diffusion 3 by installing the model in the gallery (stable-diffusion-3-medium) or by placing this YAML file in the model folder:

backend: diffusers
diffusers:
  cuda: true
  enable_parameters: negative_prompt,num_inference_steps
  pipeline_type: StableDiffusion3Pipeline
f16: false
name: sd3
parameters:
  model: v2ray/stable-diffusion-3-medium-diffusers
step: 25

You can try then generating an image:

http://localhost:9091/v1/images/generations -H "Content-Type: application/json" -d '{
  "prompt": "A cute baby sea otter", "model": "sd3"
}

Example result:

b64514236520

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

📖 Documentation and examples

👒 Dependencies

Other Changes

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.17.0...v2.17.1

LocalAI - v2.17.0

Published by mudler 4 months ago

local-ai-release-2 17-shadow
Ahoj! this new release of LocalAI comes with tons of updates, and enhancements behind the scenes!

🌟 Highlights TLDR;

  • Automatic identification of GGUF models
  • New WebUI page to talk with an LLM!
  • https://models.localai.io is live! 🚀
  • Better arm64 and Apple silicon support
  • More models to the gallery!
  • New quickstart installer script
  • Enhancements to mixed grammar support
  • Major improvements to transformers
  • Linux single binary now supports rocm, nvidia, and intel

🤖 Automatic model identification for llama.cpp-based models

Just drop your GGUF files into the model folders, and let LocalAI handle the configurations. YAML files are now reserved for those who love to tinker with advanced setups.

🔊 Talk to your LLM!

Introduced a new page that allows direct interaction with the LLM using audio transcription and TTS capabilities. This feature is so fun - now you can just talk with any LLM with a couple of clicks away.
Screenshot from 2024-06-08 12-44-41

🍏 Apple single-binary

Experience enhanced support for the Apple ecosystem with a comprehensive single-binary that packs all necessary libraries, ensuring LocalAI runs smoothly on MacOS and ARM64 architectures.

ARM64

Expanded our support for ARM64 with new Docker images and single binary options, ensuring better compatibility and performance on ARM-based systems.

Note: currently we support only arm core images, for instance: localai/localai:master-ffmpeg-core, localai/localai:latest-ffmpeg-core, localai/localai:v2.17.0-ffmpeg-core.

🐞 Bug Fixes and small enhancements

We’ve ironed out several issues, including image endpoint response types and other minor problems, boosting the stability and reliability of our applications. It is now also possible to enable CSRF when starting LocalAI, thanks to @dave-gray101.

🌐 Models and Galleries

Enhanced the model gallery with new additions like Mirai Nova, Mahou, and several updates to existing models ensuring better performance and accuracy.

Now you can check new models also in https://models.localai.io, without running LocalAI!

Installation and Setup

A new install.sh script is now available for quick and hassle-free installations, streamlining the setup process for new users.

curl https://localai.io/install.sh | sh

Installation can be configured with Environment variables, for example:

curl https://localai.io/install.sh | VAR=value sh

List of the Environment Variables:

  • DOCKER_INSTALL: Set to "true" to enable the installation of Docker images.
  • USE_AIO: Set to "true" to use the all-in-one LocalAI Docker image.
  • API_KEY: Specify an API key for accessing LocalAI, if required.
  • CORE_IMAGES: Set to "true" to download core LocalAI images.
  • PORT: Specifies the port on which LocalAI will run (default is 8080).
  • THREADS: Number of processor threads the application should use. Defaults to the number of logical cores minus one.
  • VERSION: Specifies the version of LocalAI to install. Defaults to the latest available version.
  • MODELS_PATH: Directory path where LocalAI models are stored (default is /usr/share/local-ai/models).

We are looking into improving the installer, and as this is a first iteration any feedback is welcome! Open up an issue if something doesn't work for you!

Enhancements to mixed grammar support

Mixed grammar support continues receiving improvements behind the scenes.

🐍 Transformers backend enhancements

  • Temperature = 0 correctly handled as greedy search
  • Handles custom words as stop words
  • Implement KV cache
  • Phi 3 no more requires trust_remote_code: true flag

Shout-out to @fakezeta for these enhancements!

Install models with the CLI

Now the CLI can install models directly from the gallery. For instance:

local-ai run <model_name_in gallery>

This command ensures the model is installed in the model folder at startup.

🐧 Linux single binary now supports rocm, nvidia, and intel

Single binaries for Linux now contain Intel, AMD GPU, and NVIDIA support. Note that you need to install the dependencies separately in the system to leverage these features. In upcoming releases, this requirement will be handled by the installer script.

📣 Let's Make Some Noise!

A gigantic THANK YOU to everyone who’s contributed—your feedback, bug squashing, and feature suggestions are what make LocalAI shine. To all our heroes out there supporting other users and sharing their expertise, you’re the real MVPs!

Remember, LocalAI thrives on community support—not big corporate bucks. If you love what we're building, show some love! A shoutout on social (@LocalAI_OSS and @mudler_it on twitter/X), joining our sponsors, or simply starring us on GitHub makes all the difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Thanks a ton, and.. enjoy this release!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

🧠 Models

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.16.0...v2.17.0

LocalAI - v2.16.0

Published by mudler 5 months ago

local-ai-release-2 16

Welcome to LocalAI's latest update!

🎉🎉🎉 woot woot! So excited to share this release, a lot of new features are landing in LocalAI!!!!! 🎉🎉🎉

🌟 Introducing Distributed Llama.cpp Inferencing

Now it is possible to distribute the inferencing workload across different workers with llama.cpp models !

This feature has landed with https://github.com/mudler/LocalAI/pull/2324 and is based on the upstream work of @rgerganov in https://github.com/ggerganov/llama.cpp/pull/6829.

How it works: a front-end server manages the requests compatible with the OpenAI API (LocalAI) and workers (llama.cpp) are used to distribute the workload. This makes possible to run larger models split across different nodes!

How to use it

To start workers to offload the computation you can run:

local-ai llamacpp-worker <listening_address> <listening_port>

However, you can also follow the llama.cpp README and building the rpc-server (https://github.com/ggerganov/llama.cpp/blob/master/examples/rpc/README.md), which is still compatible with LocalAI.

When starting the LocalAI server, which is going to accept the API requests, you can set a list of workers IP/address by specifying the addresses with LLAMACPP_GRPC_SERVERS:

LLAMACPP_GRPC_SERVERS="address1:port,address2:port" local-ai run

At this point the workload hitting in the LocalAI server should be distributed across the nodes!

🤖 Peer2Peer llama.cpp

LocalAI is the first AI Free, Open source project offering complete, decentralized, peer2peer while private, LLM inferencing on top of the libp2p protocol. There is no "public swarm" to offload the computation, but rather empowers you to build your own cluster of local and remote machines to distribute LLM computation.

This feature leverages the ability of llama.cpp to distribute the workload explained just above and features from one of my other projects, https://github.com/mudler/edgevpn.

LocalAI builds on top of the twos, and allows to create a private peer2peer network between nodes, without the need of centralizing connections or manually configuring IP addresses: it unlocks totally decentralized, private, peer-to-peer inferencing capabilities. Works also behind different NAT-ted networks (uses DHT and mDNS as discovery mechanism).

How it works: A pre-shared token can be generated and shared between workers and the server to form a private, decentralized, p2p network.

You can see the feature in action here:

output

How to use it

  1. Start the server with --p2p:
./local-ai run --p2p
# 1:02AM INF loading environment variables from file envFile=.env
# 1:02AM INF Setting logging to info
# 1:02AM INF P2P mode enabled
# 1:02AM INF No token provided, generating one
# 1:02AM INF Generated Token:
# XXXXXXXXXXX
# 1:02AM INF Press a button to proceed

A token is displayed, copy it and press enter.

You can re-use the same token later restarting the server with --p2ptoken (or P2P_TOKEN).

  1. Start the workers. Now you can copy the local-ai binary in other hosts, and run as many workers with that token:
TOKEN=XXX ./local-ai  p2p-llama-cpp-rpc
# 1:06AM INF loading environment variables from file envFile=.env
# 1:06AM INF Setting logging to info
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"config/config.go:288","message":"connmanager disabled\n"}
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"config/config.go:295","message":" go-libp2p resource manager protection enabled"}
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"config/config.go:409","message":"max connections: 100\n"}
# 1:06AM INF Starting llama-cpp-rpc-server on '127.0.0.1:34371'
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"node/node.go:118","message":" Starting EdgeVPN network"}
# create_backend: using CPU backend
# Starting RPC server on 127.0.0.1:34371, backend memory: 31913 MB
# 2024/05/19 01:06:01 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). # See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
# {"level":"INFO","time":"2024-05-19T01:06:01.805+0200","caller":"node/node.go:172","message":" Node ID: 12D3KooWJ7WQAbCWKfJgjw2oMMGGss9diw3Sov5hVWi8t4DMgx92"}
# {"level":"INFO","time":"2024-05-19T01:06:01.806+0200","caller":"node/node.go:173","message":" Node Addresses: [/ip4/127.0.0.1/tcp/44931 /ip4/127.0.0.1/udp/33251/quic-v1/webtransport/certhash/uEiAWAhZ-W9yx2ZHnKQm3BE_ft5jjoc468z5-Rgr9XdfjeQ/certhash/uEiB8Uwn0M2TQBELaV2m4lqypIAY2S-2ZMf7lt_N5LS6ojw /ip4/127.0.0.1/udp/35660/quic-v1 /ip4/192.168.68.110/tcp/44931 /ip4/192.168.68.110/udp/33251/quic-v1/webtransport/certhash/uEiAWAhZ-W9yx2ZHnKQm3BE_ft5jjoc468z5-Rgr9XdfjeQ/certhash/uEiB8Uwn0M2TQBELaV2m4lqypIAY2S-2ZMf7lt_N5LS6ojw /ip4/192.168.68.110/udp/35660/quic-v1 /ip6/::1/tcp/41289 /ip6/::1/udp/33160/quic-v1/webtransport/certhash/uEiAWAhZ-W9yx2ZHnKQm3BE_ft5jjoc468z5-Rgr9XdfjeQ/certhash/uEiB8Uwn0M2TQBELaV2m4lqypIAY2S-2ZMf7lt_N5LS6ojw /ip6/::1/udp/35701/quic-v1]"}
# {"level":"INFO","time":"2024-05-19T01:06:01.806+0200","caller":"discovery/dht.go:104","message":" Bootstrapping DHT"}

(Note you can also supply the token via args)

At this point, you should see in the server logs messages stating that new workers are found

  1. Now you can start doing inference as usual on the server (the node used on step 1)

Interested in to try it out? As we are still updating the documentation, you can read the full instructions here https://github.com/mudler/LocalAI/pull/2343

📜 Advanced Function calling support with Mixed JSON Grammars

LocalAI gets better at function calling with mixed grammars!

With this release, LocalAI introduces a transformative capability: support for mixed JSON BNF grammars. It allows to specify a grammar for the LLM that allows to output structured JSON and free text.

How to use it:

To enable mixed grammars, you can set in the YAML configuration file function.mixed_mode = true, for example:

  function:
    # disable injecting the "answer" tool
    disable_no_action: true

    grammar:
      # This allows the grammar to also return messages
      mixed_mode: true

This feature significantly enhances LocalAI's ability to interpret and manipulate JSON data coming from the LLM through a more flexible and powerful grammar system. Users can now combine multiple grammar types within a single JSON structure, allowing for dynamic parsing and validation scenarios.

Grammars can also turned off entirely and leave the user to determine how the data is parsed from the LLM to be correctly interpretated by LocalAI to be still compliant to the OpenAI REST spec.

For example, to interpret Hermes results, one can just annotate regexes in function.json_regex_match to extract the LLM response:

  function:
    grammar:
      disable: true
    # disable injecting the "answer" tool
    disable_no_action: true
    return_name_in_function_response: true

    json_regex_match:
    - "(?s)<tool_call>(.*?)</tool_call>"
    - "(?s)<tool_call>(.*?)"
  
    replace_llm_results:
    # Drop the scratchpad content from responses
    - key: "(?s)<scratchpad>.*</scratchpad>"
      value: ""
    replace_function_results:
    # Replace everything that is not JSON array or object, just in case.
    - key: '(?s)^[^{\[]*'
      value: ""
    - key: '(?s)[^}\]]*$'
      value: ""
    # Drop the scratchpad content from responses
    - key: "(?s)<scratchpad>.*</scratchpad>"
      value: ""

Note that regex can still be used when enabling mixed grammars is enabled.

This is especially important for models which does not support grammars - such as transformers or OpenVINO models, that now can support as well function calling. As we update the docs, further documentation can be found in the PRs that you can find in the changelog below.

🚀 New Model Additions and Updates

local-ai-yi-updates

Our model gallery continues to grow with exciting new additions like Aya-35b, Mistral-0.3, Hermes-Theta and updates to existing models ensuring they remain at the cutting edge.

This release is having major enhancements on tool calling support. Besides working on making our default models in AIO images more performant - now you can try an enhanced out-of-the-box experience with function calling in the Hermes model family ( Hermes-2-Pro-Mistral and Hermes-2-Theta-Llama-3)

Our LocalAI function model!

local-ai-functioncall-model

I have fine-tuned a function call model specific to leverage entirely the grammar support of LocalAI, you can find it in the model gallery already and on huggingface

🔄 Single Binary Release: Simplified Deployment and Management

In our continuous effort to streamline the user experience and deployment process, LocalAI v2.16.0 proudly introduces a single binary release. This enhancement, thanks to @sozercan's contributions, consolidates all variants (CUDA and non-cuda releases) and dependencies into one compact executable file.

This change simplifies the installation and update processes, reduces compatibility issues, and speeds up the setup for new users and existing deployments as now binary releases are even more portable than ever!

🔧 Bug Fixes and Improvements

A host of bug fixes have been implemented to ensure smoother operation and integration. Key fixes include enhancements to the Intel build process, stability adjustments for setuptools in Python backends, and critical updates ensuring the successful build of p2p configurations.

Migrating Python Backends: From Conda to UV

LocalAI has migrated its Python backends from Conda to UV. This transition, thanks to @cryptk contributions, enhances the efficiency and scalability of our backend operations. Users will experience faster setup times and reduced complexity, streamlining the development process and making it easier to manage dependencies across different environments.

📣 Let's Make Some Noise!

A gigantic THANK YOU to everyone who’s contributed—your feedback, bug squashing, and feature suggestions are what make LocalAI shine. To all our heroes out there supporting other users and sharing their expertise, you’re the real MVPs!

Remember, LocalAI thrives on community support—not big corporate bucks. If you love what we're building, show some love! A shoutout on social (@LocalAI_OSS and @mudler_it on twitter/X), joining our sponsors, or simply starring us on GitHub makes all the difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Thanks a ton, and.. enjoy this release!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

🧠 Models

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.15.0...v2.16.0

LocalAI - v2.15.0

Published by mudler 5 months ago

local-ai-release

🎉 LocalAI v2.15.0! 🚀

Hey awesome people! I'm happy to announce the release of LocalAI version 2.15.0! This update introduces several significant improvements and features, enhancing usability, functionality, and user experience across the board. Dive into the key highlights below, and don't forget to check out the full changelog for more detailed updates.

🌍 WebUI Upgrades: Turbocharged!

🚀 Vision API Integration

The Chat WebUI now seamlessly integrates with the Vision API, making it easier for users to test image processing models directly through the browser interface - this is a very simple and hackable interface in less then 400L of code with Alpine.JS and HTMX!

output

💬 System Prompts in Chat

System prompts can be set in the WebUI chat, which guide the user through interactions more intuitively, making our chat interface smarter and more responsive.

output

🌟 Revamped Welcome Page

New to LocalAI or haven't installed any models yet? No worries! The updated welcome page now guides users through the model installation process, ensuring you're set up and ready to go without any hassle. This is a great first step for newcomers - thanks for your precious feedback!

output

🔄 Background Operations Indicator

Don't get lost with our new background operations indicator on the WebUI, which shows when tasks are running in the background.

output

🔍 Filter Models by Tag and Category

As our model gallery balloons, you can now effortlessly sift through models by tag and category, making finding what you need a breeze.

output

🔧 Single Binary Release

LocalAI is expanding into offering single binary releases, simplifying the deployment process and making it easier to get LocalAI up and running on any system.

For the moment we have condensed the builds which disables AVX and SSE instructions set. We are also planning to include cuda builds as well.

🧠 Expanded Model Gallery

This release introduces several exciting new models to our gallery, such as 'Soliloquy', 'tess', 'moondream2', 'llama3-instruct-coder' and 'aurora', enhancing the diversity and capability of our AI offerings. Our selection of one-click-install models is growing! We pick carefully model from the most trending ones on huggingface, feel free to submit your requests in a github issue, hop to our Discord or contribute by hosting your gallery, or.. even by adding models directly to LocalAI!

local-ai-gallery
local-ai-gallery-new

Want to share your model configurations and customizations? See the docs: https://localai.io/docs/getting-started/customize-model/

📣 Let's Make Some Noise!

A gigantic THANK YOU to everyone who’s contributed—your feedback, bug squashing, and feature suggestions are what make LocalAI shine. To all our heroes out there supporting other users and sharing their expertise, you’re the real MVPs!

Remember, LocalAI thrives on community support—not big corporate bucks. If you love what we're building, show some love! A shoutout on social (@LocalAI_OSS and @mudler_it on twitter/X), joining our sponsors, or simply starring us on GitHub makes all the difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Thanks a ton, and.. enjoy this release!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

🧠 Models

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.14.0...v2.15.0

LocalAI - v2.14.0

Published by mudler 6 months ago

🚀 AIO Image Update: llama3 has landed!

We're excited to announce that our AIO image has been upgraded with the latest LLM model, llama3, enhancing our capabilities with more accurate and dynamic responses. Behind the scenes uses https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF which is ready for function call, yay!

💬 WebUI enhancements: Updates in Chat, Image Generation, and TTS

Chat TTS Image gen
chatui ttsui image

Our interfaces for Chat, Text-to-Speech (TTS), and Image Generation have finally landed. Enjoy streamlined and simple interactions thanks to the efforts of our team, led by @mudler, who have worked tirelessly to enhance your experience. The WebUI interface serves as a quick way to debug and assess models loaded in LocalAI - there is much to improve, but we have now a small, hackable interface!

🖼️ Many new models in the model gallery!

local-ai-gallery

The model gallery has received a substantial upgrade with numerous new models, including Einstein v6.1, SOVL, and several specialized Llama3 iterations. These additions are designed to cater to a broader range of tasks , making LocalAI more versatile than ever. Kudos to @mudler for spearheading these exciting updates - now you can select with a couple of click the model you like!

🛠️ Robust Fixes and Optimizations

This update brings a series of crucial bug fixes and security enhancements to ensure our platform remains secure and efficient. Special thanks to @dave-gray101, @cryptk, and @fakezeta for their diligent work in rooting out and resolving these issues 🤗

✨ OpenVINO and more

We're introducing OpenVINO acceleration, and many OpenVINO models in the gallery. You can now enjoy fast-as-hell speed on Intel CPU and GPUs. Applause to @fakezeta for the contributions!

📚 Documentation and Dependency Upgrades

We've updated our documentation and dependencies to keep you equipped with the latest tools and knowledge. These updates ensure that LocalAI remains a robust and dependable platform.

👥 A Community Effort

A special shout-out to our new contributors, @QuinnPiers and @LeonSijiaLu, who have enriched our community with their first contributions. Welcome aboard, and thank you for your dedication and fresh insights!

Each update in this release not only enhances our platform's capabilities but also ensures a safer and more user-friendly experience. We are excited to see how our users leverage these new features in their projects, freel free to hit a line on Twitter or in any other social, we'd be happy to hear how you use LocalAI!

📣 Spread the word!

First off, a massive thank you (again!) to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsors can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and.. exciting times ahead with LocalAI!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

🧠 Models

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.13.0...v2.14.0

LocalAI - 🖼️ v2.13.0 - Model gallery edition

Published by mudler 6 months ago

Hello folks, Ettore here - I'm happy to announce the v2.13.0 LocalAI release is out, with many features!

Below there is a small breakdown of the hottest features introduced in this release - however - there are many other improvements (especially from the community) as well, so don't miss out the changelog!

Check out the full changelog below for having an overview of all the changes that went in this release (this one is quite packed up).

🖼️ Model gallery

This is the first release with model gallery in the webUI, you can see now a "Model" button in the WebUI which lands now in a selection of models:

output

You can choose now models between stablediffusion, llama3, tts, embeddings and more! The gallery is growing steadly and being kept up-to-date.

The models are simple YAML files which are hosted in this repository: https://github.com/mudler/LocalAI/tree/master/gallery - you can host your own repository with your model index, or if you want you can contribute to LocalAI.

If you want to contribute adding models, you can by opening up a PR in the gallery directory: https://github.com/mudler/LocalAI/tree/master/gallery.

Rerankers

I'm excited to introduce a new backend for rerankers. LocalAI now implements the Jina API (https://jina.ai/reranker/#apiform) as a compatibility layer, and you can use existing Jina clients and point to those to the LocalAI address. Behind the hoods, uses https://github.com/AnswerDotAI/rerankers.

output

You can test this by using container images with python (this does NOT work with core images) and a model config file like this, or by installing cross-encoder from the gallery in the UI:

name: jina-reranker-v1-base-en
backend: rerankers
parameters:
  model: cross-encoder

and test it with:


    curl http://localhost:8080/v1/rerank \
      -H "Content-Type: application/json" \
      -d '{
      "model": "jina-reranker-v1-base-en",
      "query": "Organic skincare products for sensitive skin",
      "documents": [
        "Eco-friendly kitchenware for modern homes",
        "Biodegradable cleaning supplies for eco-conscious consumers",
        "Organic cotton baby clothes for sensitive skin",
        "Natural organic skincare range for sensitive skin",
        "Tech gadgets for smart homes: 2024 edition",
        "Sustainable gardening tools and compost solutions",
        "Sensitive skin-friendly facial cleansers and toners",
        "Organic food wraps and storage solutions",
        "All-natural pet food for dogs with allergies",
        "Yoga mats made from recycled materials"
      ],
      "top_n": 3
    }'

Parler-tts

There is a new backend available for tts now, parler-tts. It is possible to install and configure the model directly from the gallery. https://github.com/huggingface/parler-tts

🎈 Lot of small improvements behind the scenes!

Thanks to our outstanding community, we have enhanced the performance and stability of LocalAI across various modules. From backend optimizations to front-end adjustments, every tweak helps make LocalAI smoother and more robust.

📣 Spread the word!

First off, a massive thank you (again!) to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsors can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

🧠 Models

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.12.4...V2.13.0

LocalAI - v2.12.4

Published by mudler 6 months ago

Patch release to include https://github.com/mudler/LocalAI/pull/1985

LocalAI - v2.12.3

Published by mudler 6 months ago

I'm happy to announce the v2.12.3 LocalAI release is out!

🌠 Landing page and Swagger

Ever wondered what to do after LocalAI is up and running? Integration with a simple web interface has been started, and you can see now a landing page when hitting the LocalAI front page:

Screenshot from 2024-04-07 14-43-26

You can also now enjoy Swagger to try out the API calls directly:

swagger

🌈 AIO images changes

Now the default model for CPU images is https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF - pre-configured for functions and tools API support!
If you are an Intel-GPU owner, the Intel profile for AIO images is now available too!

🚀 OpenVINO and transformers enhancements

Now there is support for OpenVINO and transformers got token streaming support now thanks to @fakezeta!

To try OpenVINO, you can use the example available in the documentation: https://localai.io/features/text-generation/#examples

🎈 Lot of small improvements behind the scenes!

Thanks for our outstanding community, we have enhanced several areas:

  • The build time of LocalAI was speed up significantly! thanks to @cryptk for the efforts in enhancing the build system
  • @thiner worked hardly to get Vision support for AutoGPTQ
  • ... and much more! see down below for a full list, be sure to star LocalAI and give it a try!

📣 Spread the word!

First off, a massive thank you (again!) to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsors can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.11.0...v2.12.3

LocalAI - v2.12.1

Published by mudler 6 months ago

I'm happy to announce the v2.12.1 LocalAI release is out!

🌠 Landing page and Swagger

Ever wondered what to do after LocalAI is up and running? Integration with a simple web interface has been started, and you can see now a landing page when hitting the LocalAI front page:

Screenshot from 2024-04-07 14-43-26

You can also now enjoy Swagger to try out the API calls directly:

swagger

🌈 AIO images changes

Now the default model for CPU images is https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF - pre-configured for functions and tools API support!
If you are an Intel-GPU owner, the Intel profile for AIO images is now available too!

🚀 OpenVINO and transformers enhancements

Now there is support for OpenVINO and transformers got token streaming support now thanks to @fakezeta!

To try OpenVINO, you can use the example available in the documentation: https://localai.io/features/text-generation/#examples

🎈 Lot of small improvements behind the scenes!

Thanks for our outstanding community, we have enhanced several areas:

  • The build time of LocalAI was speed up significantly! thanks to @cryptk for the efforts in enhancing the build system
  • @thiner worked hardly to get Vision support for AutoGPTQ
  • ... and much more! see down below for a full list, be sure to star LocalAI and give it a try!

📣 Spread the word!

First off, a massive thank you (again!) to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsors can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.11.0...v2.12.1

LocalAI - v2.12.0

Published by mudler 6 months ago

I'm happy to announce the v2.12.0 LocalAI release is out!

🌠 Landing page and Swagger

Ever wondered what to do after LocalAI is up and running? Integration with a simple web interface has been started, and you can see now a landing page when hitting the LocalAI front page:

Screenshot from 2024-04-07 14-43-26

You can also now enjoy Swagger to try out the API calls directly:

swagger

🌈 AIO images changes

Now the default model for CPU images is https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF - pre-configured for functions and tools API support!
If you are an Intel-GPU owner, the Intel profile for AIO images is now available too!

🚀 OpenVINO and transformers enhancements

Now there is support for OpenVINO and transformers got token streaming support now thanks to @fakezeta!

To try OpenVINO, you can use the example available in the documentation: https://localai.io/features/text-generation/#examples

🎈 Lot of small improvements behind the scenes!

Thanks for our outstanding community, we have enhanced several areas:

  • The build time of LocalAI was speed up significantly! thanks to @cryptk for the efforts in enhancing the build system
  • @thiner worked hardly to get Vision support for AutoGPTQ
  • ... and much more! see down below for a full list, be sure to star LocalAI and give it a try!

📣 Spread the word!

First off, a massive thank you (again!) to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsors can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI!

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.11.0...v2.12.0

LocalAI - v2.11.0

Published by mudler 7 months ago

Introducing LocalAI v2.11.0: All-in-One Images!

Hey everyone! 🎉 I'm super excited to share what we've been working on at LocalAI - the launch of v2.11.0. This isn't just any update; it's a massive leap forward, making LocalAI easier to use, faster, and more accessible for everyone.

🌠 The Spotlight: All-in-One Images, OpenAI in a box

Imagine having a magic box that, once opened, gives you everything you need to get your AI project off the ground with generative AI. A full clone of OpenAI in a box. That's exactly what our AIO images are! Designed for both CPU and GPU environments, these images come pre-packed with a full suite of models and backends, ready to go right out of the box.

Whether you're using Nvidia, AMD, or Intel, we've got an optimized image for you. If you are using CPU-only you can enjoy even smaller and lighter images.

To start LocalAI, pre-configured with function calling, llm, tts, speech to text, and image generation, just run:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-aio-cpu

## Do you have a Nvidia GPUs? Use this instead
## CUDA 11
# docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-cuda-11
## CUDA 12
# docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-cuda-12

❤️ Why You're Going to Love AIO Images:

  • Ease of Use: Say goodbye to the setup blues. With AIO images, everything is configured upfront, so you can dive straight into the fun part - hacking!
  • Flexibility: CPU, Nvidia, AMD, Intel? We support them all. These images are made to adapt to your setup, not the other way around.
  • Speed: Spend less time configuring and more time innovating. Our AIO images are all about getting you across the starting line as fast as possible.

🌈 Jumping In Is a Breeze:

Getting started with AIO images is as simple as pulling from Docker Hub or Quay and running it. We take care of the rest, downloading all necessary models for you. For all the details, including how to customize your setup with environment variables, our updated docs have got you covered here, while you can get more details of the AIO images here.

🎈 Vector Store

Thanks to the great contribution from @richiejp now LocalAI has a new backend type, "vector stores" that allows to use LocalAI as in-memory Vector DB (https://github.com/mudler/LocalAI/issues/1792). You can learn more about it here!

🐛 Bug fixes

This release contains major bugfixes to the watchdog component, and a fix to a regression introduced in v2.10.x which was not respecting --f16, --threads and --context-size to be applied as model's defaults.

🎉 New Model defaults for llama.cpp

Model defaults has changed to automatically offload maximum GPU layers if a GPU is available, and it sets saner defaults to the models to enhance the LLM's output.

🧠 New pre-configured models

You can now run llava-1.6-vicuna, llava-1.6-mistral and hermes-2-pro-mistral, see Run other models for a list of all the pre-configured models available in the release.

📣 Spread the word!

First off, a massive thank you (again!) to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsors can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI!

🔗 Links

🎁 What's More in v2.11.0?

Bug fixes 🐛

Exciting New Features 🎉

🧠 Models

📖 Documentation and examples

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.10.1...v2.11.0

LocalAI - v2.10.1

Published by mudler 7 months ago

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

Other Changes

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.10.0...v2.10.1

LocalAI - v2.10.0

Published by mudler 7 months ago

LocalAI v2.10.0 Release Notes

Excited to announce the release of LocalAI v2.10.0! This version introduces significant changes, including breaking changes, numerous bug fixes, exciting new features, dependency updates, and more. Here's a summary of what's new:

Breaking Changes 🛠

  • The trust_remote_code setting in the YAML config file of the model are now consumed for enhanced security measures also for the AutoGPTQ and transformers backend, thanks to @dave-gray101's contribution (#1799). If your model relied on the old behavior and you are sure of what you are doing, set trust_remote_code: true in the YAML config file.

Bug Fixes 🐛

  • Various fixes have been implemented to enhance the stability and performance of LocalAI:
    • SSE no longer omits empty finish_reason fields for better compatibility with the OpenAI API, fixed by @mudler (#1745).
    • Functions now correctly handle scenarios with no results, also addressed by @mudler (#1758).
    • A Command Injection Vulnerability has been fixed by @ouxs-19 (#1778).
    • OpenCL-based builds for llama.cpp have been restored, thanks to @cryptk's efforts (#1828, #1830).
    • An issue with OSX build default.metallib has been resolved, which should now allow running the llama-cpp backend on Apple arm64, fixed by @dave-gray101 (#1837).

Exciting New Features 🎉

  • LocalAI continues to evolve with several new features:
    • Ongoing implementation of the assistants API, making great progress thanks to community contributions, including an initial implementation by @christ66 (#1761).
    • Addition of diffusers/transformers support for Intel GPU - now you can generate images and use the transformer backend also on Intel GPUs, implemented by @mudler (#1746).
    • Introduction of Bitsandbytes quantization for transformer backend enhancement and a fix for transformer backend error on CUDA by @fakezeta (#1823).
    • Compatibility layers for Elevenlabs and OpenAI TTS, enhancing text-to-speech capabilities: Now LocalAI is compatible with Elevenlabs and OpenAI TTS, thanks to @mudler (#1834).
    • vLLM now supports stream: true! This feature was introduced by @golgeek (#1749).

Dependency Updates 👒

  • Our continuous effort to keep dependencies up-to-date includes multiple updates to ggerganov/llama.cpp, donomii/go-rwkv.cpp, mudler/go-stable-diffusion, and others, ensuring that LocalAI is built on the latest and most secure libraries.

Other Changes

  • Several internal changes have been made to improve the development process and documentation, including updates to integration guides, stress reduction on self-hosted runners, and more.

Details of What's Changed

Breaking Changes 🛠

Bug fixes 🐛

Exciting New Features 🎉

👒 Dependencies

Other Changes

New Contributors

Thank you to all contributors and users for your continued support and feedback, making LocalAI better with each release!

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.9.0...v2.10.0

LocalAI - v2.9.0

Published by mudler 8 months ago

This release brings many enhancements, fixes, and a special thanks to the community for the amazing work and contributions!

We now have sycl images for Intel GPUs, ROCm images for AMD GPUs,and much more:

  • You can find the AMD GPU images tags between the container images available - look for hipblas. For example, master-hipblas-ffmpeg-core. Thanks to @fenfir for this nice contribution!
  • Intel GPU images are tagged with sycl. You can find images with two flavors, sycl-f16 and sycl-f32 respectively. For example, master-sycl-f16. Work is in progress to support also diffusers and transformers on Intel GPUs.
  • Thanks to @christ66 first efforts in supporting the Assistant API were made, and we are planning to support the Assistant API! Stay tuned for more!
  • Now LocalAI supports the Tools API endpoint - it also supports the (now deprecated) functions API call as usual. We now also have support for SSE with function calling. See https://github.com/mudler/LocalAI/pull/1726 for more
  • Support for Gemma models - did you hear? Google released OSS models and LocalAI supports it already!
  • Thanks to @dave-gray101 in https://github.com/mudler/LocalAI/pull/1728 to put efforts in refactoring parts of the code - we are going to support soon more ways to interface with LocalAI, and not only restful api!

Support the project

First off, a massive thank you to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsorship program can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI! 🚀

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.8.2...v2.9.0

LocalAI - v2.8.2

Published by mudler 8 months ago

What's Changed

Bug fixes 🐛

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.8.1...v2.8.2

LocalAI - v2.8.1

Published by mudler 8 months ago

This is a patch release, mostly containing minor patches and bugfixes from 2.8.0.

Most importantly it contains a bugfix for https://github.com/mudler/LocalAI/issues/1333 which made the llama.cpp backend to get stuck in some cases where the model starts to hallucinate and fixes to the python-based backends.

Spread the word!

First off, a massive thank you to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsorship program can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome together!

Thanks a ton, and here's to more exciting times ahead with LocalAI! 🚀

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

Other Changes

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.8.0...v2.8.1

LocalAI - v2.8.0

Published by mudler 8 months ago

This release adds support for Intel GPUs, and it deprecates old ggml-based backends which are by now superseded by llama.cpp (that now supports more architectures out-of-the-box). See also https://github.com/mudler/LocalAI/issues/1651.

Images are now based on Ubuntu 22.04 LTS instead of Debian bullseye.

Intel GPUs

There are now images tagged with "sycl". There are sycl-f16 and sycl-f32 images indicating f16 or f32 support.

For example, to start phi-2 with an Intel GPU it is enough to use the container image like this:

docker run -e DEBUG=true -ti -v $PWD/models:/build/models -p 8080:8080  -v /dev/dri:/dev/dri --rm quay.io/go-skynet/local-ai:master-sycl-f32-ffmpeg-core phi-2

Note

First off, a massive thank you to each and every one of you who've chipped in to squash bugs and suggest cool new features for LocalAI. Your help, kind words, and brilliant ideas are truly appreciated - more than words can say!

And to those of you who've been heros, giving up your own time to help out fellow users on Discord and in our repo, you're absolutely amazing. We couldn't have asked for a better community.

Just so you know, LocalAI doesn't have the luxury of big corporate sponsors behind it. It's all us, folks. So, if you've found value in what we're building together and want to keep the momentum going, consider showing your support. A little shoutout on your favorite social platforms using @LocalAI_OSS and @mudler_it or joining our sponsorship program can make a big difference.

Also, if you haven't yet joined our Discord, come on over! Here's the link: https://discord.gg/uJAeKSAGDy

Every bit of support, every mention, and every star adds up and helps us keep this ship sailing. Let's keep making LocalAI awesome, together.

Thanks a ton, and here's to more exciting times ahead with LocalAI! 🚀

What's Changed

Exciting New Features 🎉

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.7.0...v2.8.0

LocalAI - v2.7.0

Published by mudler 9 months ago

This release adds support to the transformer backend for LLM as well!

For now instance you can run codellama-7b with transformers with:

docker run -ti -p 8080:8080 --gpus all localai/localai:v2.7.0-cublas-cuda12 codellama-7b

In the quickstart there are more examples available https://localai.io/basics/getting_started/#running-models.

Note: As llama.cpp is ongoing with changes that could possible cause breakage, this release does not includes changes from https://github.com/ggerganov/llama.cpp/discussions/5138 (the future versions will).

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

👒 Dependencies

Other Changes

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.6.1...v2.6.2

LocalAI - v2.6.1

Published by mudler 9 months ago

This is a patch release containing bug-fixes around parallel request support with llama.cpp models.

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.6.0...v2.6.1

LocalAI - v2.6.0

Published by mudler 9 months ago

What's Changed

Bug fixes 🐛

Exciting New Features 🎉

👒 Dependencies

Other Changes

New Contributors

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.5.1...v2.6.0

Package Rankings
Top 5.37% on Proxy.golang.org
Badges
Extracted from project README
tests Build and Release build container images Bump dependencies Artifact Hub LocalAI Star history Chart