• text-to-image API is removed from the CLI command.
• Support cancelling gRPC stream.

• Support stream close and context manager.
• API E2E tests are added.

• Hotfix: Automatically close streaming response at the end of the stream.

Now it is available to use API calls to Friendli Dedicated Endpoints.

from friendli import Friendli

client = Friendli(use_dedicated_endpoint=True)
chat = client.chat.completions.create(
    model="{endpoint_id}",
    messages=[
        {
            "role": "user",
            "content": "Give three tips for staying healthy.",
        }
    ]
)

If you want to send a request to a specific adapter of the Multi-LoRA endpoint, provide "{endpoint_id}:{adapter_route}" to model argument:

from friendli import Friendli

client = Friendli(use_dedicated_endpoint=True)
chat = client.chat.completions.create(
   model="{endpoint_id}:{adapter_route}",
   messages=[
       {
           "role": "user",
           "content": "Give three tips for staying healthy.",
       }
   ]
)

• Deprecate model conversion and quantization. Alternatively, please use friendli-model-optmizer to quantize your models.
• Increase default HTTP timeout.

• Support for Tool Calling API: Added new API to support tool calling.
• Phi3 INT8 Support: Implemented support for Phi3 INT8.
• Snowflake Arctic FP8 Quantizer: Introduced new quantizer for Snowflake Arctic FP8.
• Added support for INT8 quantization for Llama and refactored quantizer to use only safetensors.

Updating Patch Version

This patch version Introduces explicit resource management to prevent unexpected resource leaks.
By default, the library closes underlying HTTP and gRPC connections when the client is garbage-collected. However, you can now manually close the Friendli or AsyncFriendli client using the .close() method or utilize a context manager to ensure proper closure when exiting a with block.

Usage examples

import asyncio
from friendli import AsyncFriendli

client = AsyncFriendli(base_url="0.0.0.0:8000", use_grpc=True)

async def run():
    async with client:
        stream = await client.completions.create(
            prompt="Explain what gRPC is. Also give me a Python code snippet of gRPC client.",
            stream=True,
            top_k=1,
        )

        async for chunk in stream:
            print(chunk.text, end="", flush=True)

asyncio.run(run())

• gRPC client support for completions API.

• Minor: add a default value for the "index" and "text" fields of the completion stream's chunk.

• Support Phi3 FP8 conversion.
• Hotfix for safetensor checkpoint saver.

• Optimize CPU RAM usage during quantization with offloading
• Support FP8 conversion for DBRX, Mixtral, and Command R+

• Hotfix for LoRA checkpoint saving error.

New Features

• FP8 Checkpoint Conversion: We've introduced a new feature for FP8 checkpoint conversion.
• Sharded Safetensors Checkpoint Saving: Added the ability to save sharded safetensors checkpoints.
• LoRA Support on Mistral Model: We have added support for LoRA (Low-Rank Adaptation) on the Mistral model.

Bug Fixes

• BF16 Hotfix: Addressed an urgent issue with bf16 processing.
• BFloat Safetensors Conversion: Fixed an issue related to bfloat conversion for safetensors.
• Automatic Token Refresh: Resolved a bug affecting automatic token refresh.

• Add base_model_name_or_path option to friendli model convert-adapter.
• Remove stale dependencies.

• Update protobuf schema.
• Patch sending API requests with content type application/protobuf.

• Now resources of Friendli Dedicated Endpoints can be managed with CLI and SDK. The available resources are endpoint, model, team, and project.
• Login with CLI is now available. SSO login is also available.
• Update on Multi-LoRA checkpoint conversion.

Patch Version v1.2.3

• Distribute a Python package as type hinted.

Patch Version v1.2.3

• Support pydantic V1 compatibility.

Release Patch Version

• Package dependencies are updated.

Release Patch Version v1.2.1

• Update package dependencies (no more exact version matching).
• Add Mixtral model type
• Add a stop option to completions and chat completions SDK/CLI.