modelfusion

Added

• Experimental generateToolCallsOrText function for generating a multiple parallel tool call using the OpenAI chat/tools API.

Added

• ChatML prompt format.

Changed

• breaking change: ChatPrompt structure and terminology has changed to align more closely with OpenAI and similar chat prompts. This is also in preparation for integrating images and function calls results into chat prompts.
• breaking change: Prompt formats are now namespaced. Use e.g. Llama2PromptFormat.chat() instead of mapChatPromptToLlama2Format(). See Prompt Format for documentation of the new prompt formats.

Added

• Experimental generateToolCall function for generating a single tool call using the OpenAI chat/tools API.

Changed

• Refactored JSON parsing to use abstracted schemas. You can now use parseJSON and safeParseJSON to securely parse JSON objects and optionally type-check them using any schema (e.g. a Zod schema).

Added

• Ollama 0.1.9 support: format (for forcing JSON output) and raw settings
• Improved Ollama settings documentation

Added

• Support for fine-tuned OpenAI gpt-4-0613 models
• Support for trimWhitespace model setting in streamText calls

Added

• Image support for OpenAIChatMessage.user
• mapInstructionPromptToBakLLaVA1ForLlamaCppFormat prompt format

Changed

• breaking change: VisionInstructionPrompt was replaced by an optional image field in InstructionPrompt.

Added

• Support for OpenAI vision model.
  • Example: examples/basic/src/model-provider/openai/openai-chat-stream-text-vision-example.ts
• VisionInstructionPromptFormat: abstraction for single instruction, single image vision prompts. Mappings for Llama.cpp and OpenAI vision models available.
  • Example: examples/basic/src/model-provider/llamacpp/llamacpp-stream-text-vision-with-vision-prompt-example.ts

Added

• Support for OpenAI chat completion seed and responseFormat options.

Added

• OpenAI speech generation support. Shoutout to @bjsi for the awesome contribution!

Added

• OpenAI gpt-3.5-turbo-1106, gpt-4-1106-preview, gpt-4-vision-preview chat models.
• OpenAI Dalle-E-3 image model.

Changed

• breaking change: OpenAIImageGenerationModel now requires a model parameter.

Added

• Support image input for multi-modal Llama.cpp models (e.g. Llava, Bakllava).

Changed

• breaking change: Llama.cpp prompt format has changed to support images. Use .withTextPrompt() to get a text prompt format.

Added

• ElevenLabs eleven_turbo_v2 support.

Fixed

• breaking change: Uncaught errors were caused by custom Promises. ModelFusion now uses only standard Promises. To get full responses from model function, you now need to use the { returnType: "full" } option instead of calling .asFullResponse() on the result.

Improved

• ModelFusion server error logging and reporting.

Fixed

• ModelFusion server creates directory for runs automatically when errors are thrown.

Added

• Support for Cohere v3 embeddings.