End-to-end platform for building voice first multimodal agents
MIT License
Bolna is the end-to-end open source production ready framework for quickly building LLM based voice driven conversational applications.
https://github.com/bolna-ai/bolna/assets/1313096/2237f64f-1c5b-4723-b7e7-d11466e9b226
Bolna helps you create AI Voice Agents which can be instructed to do tasks beginning with:
Twilio
, Plivo
, Exotel
, etc.Deepgram
, etc.OpenAI
, Llama
, Cohere
, Mistral
, etc to handle conversationsAWS Polly
, XTTS
, ElevenLabs
, Deepgram
etc.Refer to the docs for a deepdive into all supported providers.
A basic local setup includes usage of Twilio or Plivo for telephony. We have dockerized the setup in local_setup/
. One will need to populate an environment .env
file from .env.sample
.
The setup consists of four containers:
ngrok
: for tunneling. One will need to add the authtoken
to ngrok-config.yml
redis
: for persisting agents & prompt dataUse docker to build the images using .env
file as the environment file and run them locally
docker-compose build --no-cache <twilio-app | plivo-app>
: rebuild imagesdocker-compose up <twilio-app | plivo-app>
: run the build imagesOnce the docker containers are up, you can now start to create your agents and instruct them to initiate calls.
Once you have the above docker setup and running, you can create agents and initiate calls.
http://localhost:5001/agent
{
"agent_config": {
"agent_name": "Alfred",
"agent_type": "other",
"agent_welcome_message": "Welcome",
"tasks": [
{
"task_type": "conversation",
"toolchain": {
"execution": "parallel",
"pipelines": [
[
"transcriber",
"llm",
"synthesizer"
]
]
},
"tools_config": {
"input": {
"format": "pcm",
"provider": "twilio"
},
"llm_agent": {
"agent_flow_type": "streaming",
"provider": "openai",
"request_json": true,
"model": "gpt-3.5-turbo-16k",
"use_fallback": true
},
"output": {
"format": "pcm",
"provider": "twilio"
},
"synthesizer": {
"audio_format": "wav",
"provider": "elevenlabs",
"stream": true,
"provider_config": {
"voice": "Meera - high quality, emotive",
"model": "eleven_turbo_v2_5",
"voice_id": "TTa58Hl9lmhnQEvhp1WM"
},
"buffer_size": 100.0
},
"transcriber": {
"encoding": "linear16",
"language": "en",
"provider": "deepgram",
"stream": true
}
},
"task_config": {
"hangup_after_silence": 30.0
}
}
]
},
"agent_prompts": {
"task_1": {
"system_prompt": "Ask if they are coming for party tonight"
}
}
}
agent_id
. Use this agent_id
to initiate a call via the telephony server running on 8001
port (for Twilio) or 8002
port (for Plivo) at http://localhost:8001/call
{
"agent_id": "4c19700b-227c-4c2d-8bgf-42dfe4b240fc",
"recipient_phone_number": "+19876543210",
}
You can populate the .env
file to use your own keys for providers.
Provider | Environment variable to be added in .env file |
---|---|
Deepgram | DEEPGRAM_AUTH_TOKEN |
These are the current supported LLM Provider Family: https://github.com/bolna-ai/bolna/blob/477e08d6800dbf02931abeeea883d78451b7d7e2/bolna/providers.py#L29-L44
For LiteLLM based LLMs, add either of the following to the .env
file depending on your use-case:
LITELLM_MODEL_API_KEY
: API Key of the LLM
LITELLM_MODEL_API_BASE
: URL of the hosted LLM
LITELLM_MODEL_API_VERSION
: API VERSION for LLMs like Azure
For LLMs hosted via VLLM, add the following to the .env
file:
VLLM_SERVER_BASE_URL
: URL of the hosted LLM using VLLM
Provider | Environment variable to be added in .env file |
---|---|
AWS Polly | Accessed from system wide credentials via ~/.aws |
Elevenlabs | ELEVENLABS_API_KEY |
OpenAI | OPENAI_API_KEY |
Deepgram | DEEPGRAM_AUTH_TOKEN |
Provider | Environment variable to be added in .env file |
---|---|
Twilio |
TWILIO_ACCOUNT_SID , TWILIO_AUTH_TOKEN , TWILIO_PHONE_NUMBER
|
Plivo |
PLIVO_AUTH_ID , PLIVO_AUTH_TOKEN , PLIVO_PHONE_NUMBER
|
In case you wish to extend and add some other Telephony like Vonage, Telnyx, etc. following the guidelines below:
Though the repository is completely open source, you can connect with us if interested in managed hosted offerings or more customized solutions.
We love all types of contributions: whether big or small helping in improving this community resource.