ChatdollKit

ChatdollKit enables you to make your 3D model into a chatbot

APACHE-2.0 License

Stars
723
Committers
2
View Code on GitHub View on X
Ecosystems: Unity, Azure, ChatGPT
ChatdollKit - v0.8.0 Latest Release

Published by uezo about 2 months ago

💎 What's New in Version 0.8 Beta

To run the demo for version 0.8.0 beta, please follow the steps below after importing the dependencies:

  • Open scene Demo/Demo08.
  • Select AIAvatarVRM object in scene.
  • Set OpenAI API key to following components on inspector:
    • ChatGPTService
    • OpenAITTSLoader
    • OpenAISpeechListener
  • Run on Unity Editor.
  • Say "こんにちは" or word longer than 3 characters.
  • Enjoy👍

⚡ Optimized AI Dialog Processing

We've boosted response speed with parallel processing and made it easier for you to customize behavior with your own code. Enjoy faster, more flexible AI conversations!

🥰 Emotionally Rich Speech

Adjusts vocal tone dynamically to match the conversation, delivering more engaging and natural interactions.

🎤 Enhanced Microphone Control

Microphone control is now more flexible than ever! Easily start/stop devices, mute/unmute, and adjust voice recognition thresholds independently.

🍩 Other Changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/0.7.7...0.8.0

ChatdollKit - v0.7.7

Published by uezo about 2 months ago

🥰 Support StyleBertVits2

We've added support for Text-to-Speech using the StyleBertVits2 API! 🎙️✨ Now, your AI characters can speak with even more expressive and dynamic voices, making them shine brighter than ever! 😎 Get ready to take your character's charm to the next level! 🚀💫

💕 Support Cohere Command R 💕

🐸 Other Changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/0.7.6...v0.7.7

ChatdollKit - v0.7.6

Published by uezo 3 months ago

What's Changed

🎓LLM related updates

🗣️ Dialog control

🥰 3D model control

🐈 Others

Full Changelog: https://github.com/uezo/ChatdollKit/compare/0.7.5...0.7.6

ChatdollKit - v0.7.5

Published by uezo 4 months ago

Dify Support 💙

Other changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/0.7.4...0.7.5

ChatdollKit - v0.7.4

Published by uezo 4 months ago

👀 Enhanced Vision Capabilities

This update introduces autonomous vision input for Gemini and Claude, and adds vision input support for WebGL. Now, various AIs can offer richer conversational experiences with integrated vision input across different platforms.

Full Changelog: https://github.com/uezo/ChatdollKit/compare/0.7.3...0.7.4

ChatdollKit - v0.7.3

Published by uezo 4 months ago

👀 Support dynamic vision input for ChatGPT

By adding a SimpleCamera to the scene and including [vision:camera] in the response message, the system will autonomously capture images when visual input is required for a response.

📦 Easy setup by modularized UI components

Microphone volume sliders and request input forms have been modularized. These can now be used immediately by simply adding the prefabs to the scene without any additional setup.

🎙️ dB-based microphone volume

✨ Other changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.7.2...0.7.3

ChatdollKit - v0.7.2

Published by uezo 7 months ago

🖼️ Support vision

Set image to request message as payload for multimodal conversation.

🚀 Configuration-free demo

Just start without any configurations. Set API key (and others if you want) at runtime.

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.7.1...v0.7.2

ChatdollKit - v0.7.1

Published by uezo 10 months ago

🥰😇 Change avatar on runtime

Support changing avatar on runtime. Use ModelController.SetAvatar to change the avatar to another on runtime.
If you want to try it on editor, set another avatar on the scene to the inspector of ModelController and push Change Avatar button(appears runtime only).

🎙️ Better microphone management

Mute microphone when not listening for WakeWord or VoiceRequest.
Also added IsMuted property to DialogController so that you can control mute/unmute manually. Default is false.

  • IsMuted == false: Microphone will be on when WakeWordListener or VoiceRequestProvider is listening.
  • IsMuted == true: Microphone will be off.

You can change the value manually on inspector or in script.

🍱 Other changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.7.0...v0.7.1

ChatdollKit - v0.7.0

Published by uezo 10 months ago

🤖 LLM-based Dialog Processing

✅Multiple LLMs: ChatGPT / Azure OpenAI Service, Anthropic Claude, Google Gemini Pro and others
✅Agents: Function Calling (ChatGPT / Gemini) or your prompt engineering
✅Multimodal: GPT-4V and Gemini-Pro-Vision are suppored
✅Emotions: Autonomous face expression and animation

We've developed a versatile framework that standardizes the processes of Routing, Chatting (including facial expressions and motions), and executing Tools using various Large Language Models (LLMs). This framework allows for easy customization by simply swapping out LLM-specific components.

Additionally, you can support any LLM by creating your own components that implement the ILLMService interface.

To use this new LLM-based dialog, attach LLMRouter, LLMContentSkill and ILLMService component like ChatGPTService in ChatdollKit.LLM package. Also attach function skills that extends LLMFunctionSkillBase if you build an AI agent. See DemoChatGPT that works right out-of-the-box🎁.

NOTE: ChatdollKit.Dialog.Processor.ChatGPT* components are deprecated.

🐉 Other Changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6.6...v0.7.0

ChatdollKit - v0.6.6

Published by uezo 11 months ago

🐘 Improve stability

Previously, ChatGPT (OpenAI API) sometimes does not return a response, which prevents the character from continuing the conversation.😔
However, in this update, the system will automatically retry in the absence of a response, allowing the conversation to continue.

💃 Autonomous animation

Set animations before starting conversation. (e.g. at Start())

chatGPTContentSkill.RegisterAnimation("waving_arm", new Model.Animation("BaseParam", 10, 3.0f));
chatGPTContentSkill.RegisterAnimation("look_away", new Model.Animation("BaseParam", 6, 3.0f, "AGIA_Layer_look_away_01", "Additive Layer"));

List them in the prompt and provide a guide on how to use them.

* You can express your emotions through the following animations:

- waving_arm
- look_away

* If you want to express emotions with gestures, insert the animation into the response message like [anim:waving_arm].

😴 Multiple idling mode

You can have multiple idling mode. The default is normal. Add animations and face expressions if you want to add another mode like below:

modelController.AddIdleAnimation(new Model.Animation("BaseParam", 101, 5f), mode: "sleep");
modelController.AddIdleFace("sleep", "Blink");

Switch using ChangeIdlingModeAsync.

modelController.ChangeIdlingModeAsync("sleep");

The animation for sleep starts immediately.

🍩 Other updates

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6.5...v0.6.6

ChatdollKit - v0.6.5

Published by uezo 12 months ago

🌊 Azure OpenAI Service

To use Azure OpenAI Service set following info on inspector of ChatGPTService component:

  1. Endpoint url with configurations to Chat Completion Url 
format: https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}

  1. API Key to Api Key

  2. Set true to Is Azure

NOTE: Model on inspector is ignored. Engine in url is used.

🗣️ OpenAI Speech Services

Support multiple languages and allow you to switch between them without changing configurations.

NOTE: Text-to-Speech doesn't support WebGL.

🚰 Real-time stream speech recognition

Use AzureStreamVoiceRequestProvider to perform real-time speech recognition.
This component depends on Azure Speech SDK.

NOTE: Microphone volume slider doesn't support controlling this component.

☺️ Other small changes

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6.4...v0.6.5

ChatdollKit - v0.6.4

Published by uezo about 1 year ago

🥰 Make Unagirl more kawaii

By applying eye makeup, I enhanced the attractiveness of her face.🪄

  • Enhanced the double eyelids.
  • Increased the volume of the lower eyelashes.

🐕 Support Retrieval Augmented Generation

You can create RAG-based AI agent just by adding ChatGPT function skill. See RetrievalQASkill example that can answer based on OpenAI terms of use.

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6.3...v0.6.4

ChatdollKit - v0.6.3

Published by uezo about 1 year ago

Feature updates⚡️

Minor updates but they are necessary for creating apps.

Unagirl, a new demo character debuts🥰

See Unagirl in dress at vroid hub https://hub.vroid.com/characters/7204218679890800631/models/6961078537805052952

Other updates and fixes🐈

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6.2...v0.6.3

ChatdollKit - v0.6.2

Published by uezo about 1 year ago

🥳Create ChatGPT-based AI assistant smarter

ChatGPT is no longer just one of the various skills but is integrated as a conversation processing system of ChatdollKit.

You can add skills much smarter: All you have to do is making the definitions of the interface of ChatGPT function calling and to implement the logic to handle the result from the function.

See also the example.
https://github.com/uezo/ChatdollKit/blob/master/Examples/ChatGPT/FunctionSkills/WeatherSkill.cs

🌎 Better experience on WebGL

👍 Other updates

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6.1...v0.6.2

ChatdollKit - v0.6.1

Published by uezo over 1 year ago

⚡️ Improve performance, stability and developer experience

🐞 Fix bug

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.6...v0.6.1

ChatdollKit - v0.6

Published by uezo over 1 year ago

❤️ Emotion Simulation powered by ChatGPT

Add ChatGPTEmotionSkill as an out-of-the-box example that express the emotion as face expressions while conversation.
Also add sample motions while thinking (waiting response from ChatGPT) and support multiple idle animations (change pose periodically).
You can build rich 3D AI virtual assistant lightning fast⚡️

💡 Improve prompt engineering interface

Support one-shot prompt: system message content and a pair of user and assistant message content as an example turn. You can configure it on inspector, and even runtime for they are the public fields.

🚅 Support runtime VRM loading

Support configuration at runtime. This enables you to build apps that uses user own VRM.

⚡️ Improve performance and stability

Improve FPS about 30% on my machine and stability of controlling animation gets better. They are basically internal refactoring but that includes some destructive changes.

  • Use animator parameters to set animation instead of state name.
// ~ v0.5.3
response.AddAnimation("AnimationName");

// v0.6
response.AddAnimation("BaseParam", 6);
  • PreGap and PostGap no longer available for the argument of Animation and FaceExpression. Use duration to adjust timing.

Other changes

New Contributors

Full Changelog: https://github.com/uezo/ChatdollKit/compare/v0.5.3...v0.6

ChatdollKit - v0.5.3

Published by uezo over 1 year ago

ChatGPT support✨

Skill for contextual conversation with ChatGPT and it's demo is included!
You can develop and start talking 3D model based virtual agent lightning fast⚡️

  • Add ChatGPT ready example and demo #198

Add UI components😊

Add slider for microphone volume and FPS indicator.

  • Add UI Components (Mic volume and FPS) #199

Others

Fix some bugs for WebGL platform.

  • Fix some build errors #197
ChatdollKit - v0.5.2

Published by uezo almost 2 years ago

Separate ChatdollKit components from avatar object

Separate ChatdollKit components from avatar object to manage components easier; setting up, changing 3D model or something like that use cases. At this version, the developer can complete configuration by just adding ChatdollKit prefab to the hierarchy and configuring some components attached to this prefab. https://github.com/uezo/ChatdollKit/pull/190

Support uLipSync configuration

To set up uLipSync select uLipSync on the inspector of ModelController and then start Setup ModelController. https://github.com/uezo/ChatdollKit/pull/191

Demo

ChatdollKit - v0.5.1

Published by uezo over 2 years ago

This update includes small/internal changes, bugfixes and Watson Assisntant support.

  • Add support for implementing prompter in RequestProcessor #182
  • Add support for Watson Assistant #184
  • Fix bug sending ping to remote dialog server before URL is set #183
  • Fix index error in Watson TTSLoader #185
ChatdollKit - v0.5.0

Published by uezo over 2 years ago

Support server side dialog management

  • Devide request processing from conversation loop #169

Improve editor

  • Switch speech services and request processor on editor #177
  • Make it possible to send text to WakeWordListener instead of voice #170
  • Move dialog configurations to DialogController #171

Other updates

  • Fix bug for updating face clip fails #168
  • Make constructors of Request and Response to take more arguments #172
  • Change file structure
Badges
Extracted from project README
Related Projects
speech-to-element

A simple way to add speech to text functionality to your website

24 Jun 2023 11
ollamazure

Emulates Azure OpenAI API on your local machine using Ollama and open-source models.

03 Jul 2024 19