modelfusion

Added

• Multi-modal chat prompts. Supported by OpenAI vision chat models and by BakLLaVA prompt format.

Changed

• breaking change: renamed ChatPrompt to TextChatPrompt to distinguish it from multi-modal chat prompts.

Added

• experimental: modelfusion/extension export with functions and classes that are necessary to implement providers in 3rd party node modules. See lgrammel/modelfusion-example-provider for an example.

Added

• OpenAIChatMessage function call support.

Added

• Support for OpenAI-compatible chat APIs. See OpenAI Compatible for details.

  import {
    BaseUrlApiConfiguration,
    openaicompatible,
    generateText,
  } from "modelfusion";
  
  const text = await generateText(
    openaicompatible
      .ChatTextGenerator({
        api: new BaseUrlApiConfiguration({
          baseUrl: "https://api.fireworks.ai/inference/v1",
          headers: {
            Authorization: `Bearer ${process.env.FIREWORKS_API_KEY}`,
          },
        }),
        model: "accounts/fireworks/models/mistral-7b",
      })
      .withTextPrompt(),
  
    "Write a story about a robot learning to love"
  );
  

Added

• Introduce uncheckedSchema() facade function as an easier way to create unchecked ModelFusion schemas. This aligns the API with zodSchema().

Changed

• breaking change: Renamed InstructionPrompt interface to MultiModalInstructionPrompt to clearly distinguish it from TextInstructionPrompt.
• breaking change: Renamed .withBasicPrompt methods for image generation models to .withTextPrompt to align with text generation models.

Added

• Introduce zodSchema() function as an easier way to create new ModelFusion Zod schemas. This clearly distinguishes it from ZodSchema that is also part of the zod library.

breaking change: generateStructure and streamStructure redesign. The new API does not require function calling and StructureDefinition objects any more. This makes it more flexible and it can now be used in 3 ways:

• with OpenAI function calling:

  const model = openai
    .ChatTextGenerator({ model: "gpt-3.5-turbo" })
    .asFunctionCallStructureGenerationModel({
      fnName: "...",
      fnDescription: "...",
    });
  

• with OpenAI JSON format:

  const model = openai
    .ChatTextGenerator({
      model: "gpt-4-1106-preview",
      temperature: 0,
      maxCompletionTokens: 1024,
      responseFormat: { type: "json_object" },
    })
    .asStructureGenerationModel(
      jsonStructurePrompt((instruction: string, schema) => [
        OpenAIChatMessage.system(
          "JSON schema: \n" +
            JSON.stringify(schema.getJsonSchema()) +
            "\n\n" +
            "Respond only using JSON that matches the above schema."
        ),
        OpenAIChatMessage.user(instruction),
      ])
    );
  

• with Ollama (and a capable model, e.g., OpenHermes 2.5):

  const model = ollama
    .TextGenerator({
      model: "openhermes2.5-mistral",
      maxCompletionTokens: 1024,
      temperature: 0,
      format: "json",
      raw: true,
      stopSequences: ["\n\n"], // prevent infinite generation
    })
    .withPromptFormat(ChatMLPromptFormat.instruction())
    .asStructureGenerationModel(
      jsonStructurePrompt((instruction: string, schema) => ({
        system:
          "JSON schema: \n" +
          JSON.stringify(schema.getJsonSchema()) +
          "\n\n" +
          "Respond only using JSON that matches the above schema.",
        instruction,
      }))
    );
  

See generateStructure for details on the new API.

Changed

• breaking change: Restructured multi-modal instruction prompts and OpenAIChatMessage.user()

Added

• Multi-tool usage from open source models

  Use TextGenerationToolCallsOrGenerateTextModel and related helper methods .asToolCallsOrTextGenerationModel() to create custom prompts & parsers.

  Examples:

  • examples/basic/src/model-provider/ollama/ollama-use-tools-or-generate-text-openhermes-example.ts
  • examples/basic/src/model-provider/llamacpp/llamacpp-use-tools-or-generate-text-openhermes-example.ts

  Example prompt format:

  • examples/basic/src/tool/prompts/open-hermes.ts for OpenHermes 2.5

Removed

• breaking change: Removed FunctionListToolCallPromptFormat. See examples/basic/src/model-provide/ollama/ollama-use-tool-mistral-example.ts for how to implement a ToolCallPromptFormat for your tool.

Changed

• breaking change: Rename Speech to SpeechGenerator in facades
• breaking change: Rename Transcription to Transcriber in facades

Added

• Anthropic Claude 2.1 support

Introducing model provider facades:

const image = await generateImage(
  openai.ImageGenerator({ model: "dall-e-3", size: "1024x1024" }),
  "the wicked witch of the west in the style of early 19th century painting"
);

Added

• Model provider facades. You can e.g. now use ollama.TextGenerator(...) instead of new OllamaTextGenerationModel(...).

Changed

• breaking change: Fixed method name isParallizable to isParallelizable in EmbeddingModel.

Removed

• breaking change: removed HuggingFaceImageDescriptionModel. Image description models will be replaced by multi-modal vision models.

Improved

• Increase OpenAI chat streaming resilience.

Prompt format and tool calling improvements.

Added

• text prompt format. Use simple text prompts, e.g. with OpenAIChatModel:

  const textStream = await streamText(
    new OpenAIChatModel({
      model: "gpt-3.5-turbo",
    }).withTextPrompt(),
    "Write a short story about a robot learning to love."
  );
  

• .withTextPromptFormat to LlamaCppTextGenerationModel for simplified prompt construction:

  const textStream = await streamText(
    new LlamaCppTextGenerationModel({
      // ...
    }).withTextPromptFormat(Llama2PromptFormat.text()),
    "Write a short story about a robot learning to love."
  );
  

• FunctionListToolCallPromptFormat to simplify tool calls with text models

• .asToolCallGenerationModel() to OllamaTextGenerationModel to simplify tool calls:

  const { tool, args, toolCall, result } = await useTool(
    new OllamaTextGenerationModel({
      model: "mistral",
      temperature: 0,
    }).asToolCallGenerationModel(FunctionListToolCallPromptFormat.text()),
    calculator,
    "What's fourteen times twelve?"
  );
  

Improved

• better error reporting when using exponent backoff retries

Removed

• breaking change: removed input from InstructionPrompt (was Alpaca-specific, AlpacaPromptFormat still supports it)

Remove section newlines from Llama 2 prompt format.

Ollama edge case and error handling improvements.

Breaking change: the tool calling API has been reworked to support multiple parallel tool calls. This required multiple breaking changes (see below). Check out the updated tools documentation for details.

Changed

• Tool now has parameters and returnType schemas (instead of inputSchema and outputSchema).
• useTool uses generateToolCall under the hood. The return value and error handling has changed.
• useToolOrGenerateText has been renamed to useToolsOrGenerateText. It now uses generateToolCallsOrText under the hood. The return value and error handling has changed. It can now invoke several tools in parallel and returns an array of tool results.
• The maxRetries parameter in guard has been replaced by a maxAttempt parameter.

Removed

• generateStructureOrText has been removed.