Friendli: the fastest serving engine for generative AI
APACHE-2.0 License
Bot releases are visible (Hide)
Published by kooyunmo about 2 months ago
text-to-image
API is removed from the CLI command.Published by kooyunmo about 2 months ago
Published by kooyunmo 2 months ago
Now it is available to use API calls to Friendli Dedicated Endpoints.
from friendli import Friendli
client = Friendli(use_dedicated_endpoint=True)
chat = client.chat.completions.create(
model="{endpoint_id}",
messages=[
{
"role": "user",
"content": "Give three tips for staying healthy.",
}
]
)
If you want to send a request to a specific adapter of the Multi-LoRA endpoint, provide "{endpoint_id}:{adapter_route}" to model
argument:
from friendli import Friendli
client = Friendli(use_dedicated_endpoint=True)
chat = client.chat.completions.create(
model="{endpoint_id}:{adapter_route}",
messages=[
{
"role": "user",
"content": "Give three tips for staying healthy.",
}
]
)
Published by kooyunmo 3 months ago
friendli-model-optmizer
to quantize your models.Published by kooyunmo 3 months ago
Published by kooyunmo 4 months ago
This patch version Introduces explicit resource management to prevent unexpected resource leaks.
By default, the library closes underlying HTTP and gRPC connections when the client is garbage-collected. However, you can now manually close the Friendli
or AsyncFriendli
client using the .close()
method or utilize a context manager to ensure proper closure when exiting a with
block.
import asyncio
from friendli import AsyncFriendli
client = AsyncFriendli(base_url="0.0.0.0:8000", use_grpc=True)
async def run():
async with client:
stream = await client.completions.create(
prompt="Explain what gRPC is. Also give me a Python code snippet of gRPC client.",
stream=True,
top_k=1,
)
async for chunk in stream:
print(chunk.text, end="", flush=True)
asyncio.run(run())
Published by kooyunmo 4 months ago
Published by kooyunmo 4 months ago
Published by kooyunmo 4 months ago
Published by kooyunmo 5 months ago
Published by kooyunmo 7 months ago
Published by kooyunmo 7 months ago
Published by kooyunmo 7 months ago
base_model_name_or_path
option to friendli model convert-adapter
.Published by kooyunmo 7 months ago
application/protobuf
.Published by kooyunmo 7 months ago
endpoint
, model
, team
, and project
.Published by kooyunmo 8 months ago
Published by kooyunmo 8 months ago
pydantic
V1 compatibility.Published by kooyunmo 8 months ago
Published by kooyunmo 8 months ago
stop
option to completions and chat completions SDK/CLI.