kani

Improvements

• Added a warning in Kani.chat_round to use Kani.full_round when AI functions are defined
• Added examples in Google Colab
• Other documentation improvements

Fixes

• Fixed an issue where the ctransformers engine could overrun its context length (e.g. see https://github.com/zhudotexe/kani/actions/runs/6152842183/job/16695721588)

Improvements

• Made chat_in_terminal work in Google Colab, rather than having to use await chat_in_terminal_async

• HuggingFace Engine: Fixed an issue where completion message lengths were overreported by an amount equal to the prompt length.
• Other documentation improvements

Improvements

• Added Kani.add_to_history, a method that is called whenever kani adds a new message to the chat context
• httpclient.BaseClient.request now returns a Response to aid low-level implementation
  • .get() and .post() are unchanged
• Add additional documentation about GPU support for local models
• Other documentation improvements

Improvements

• Engines: Added Engine.function_token_reserve() to dynamically reserve a number of tokens for a function list
• OpenAI: The OpenAIEngine now reads the OPENAI_API_KEY environment variable by default if no api key or client is specified
• Documentation improvements (polymorphism, mixins, extension packages)

BREAKING CHANGES

These should hopefully be the last set of breaking changes until v1.0. We're finalizing some of the attribute names for clarity and publication.

• renamed Kani.always_include_messages to Kani.always_included_messages

Features & Improvements

• @ai_functions with synchronous signatures now run in a thread pool in order to prevent blocking the asyncio event loop
• OpenAI: Added the ability to specify the API base and additional headers (e.g. for proxy APIs).
• Various documentation improvements

BREAKING CHANGES

• Renamed Kani.get_truncated_chat_history to Kani.get_prompt

Additions & Improvements

• Added CTransformersEngine and LlamaCTransformersEngine (thanks @Maknee!)
• Added a lower-level Kani.get_model_completion to make a prediction at the current chat state (without modifying the chat history)
• Added the auto_truncate param to @ai_function to opt in to kani trimming long responses from a function (i.e., responses that do not fit in a model's context)
• Improved the internal handling of tokens when the chat history is directly modified
• ChatMessage.[role]() classmethods now pass kwargs to the constructor
• LLaMA: Improved the fidelity of non-strict-mode LLaMA prompting
• OpenAI: Added support for specifying an OpenAI organization and configuring retry
• Many documentation improvements

Fixes

• OpenAI message length could return too short on messages with no content
• Other minor fixes and improvements

• Add chat_in_terminal_async for async environments (e.g. Google Colab)
• Add quickstart Colab notebook

Initial release!