axflow

import { StreamToIterable } from '@axflow/models/shared';
import { GoogleGenerateContent } from '@axflow/models/google/generate-content';

const stream = await GoogleGenerateContent.stream(
  {
    model: 'gemini-pro',
    contents: [
      {
        parts: [
          {
            text: 'Write a two sentence story about a magic backpack',
          },
        ],
      },
    ],
  },
  {
    apiKey: process.env.GOOGLE_API_KEY,
  }
);

for await (const chunk of StreamToIterable(stream)) {
  console.log(chunk);
}

For example, we can run Llama2 70B on Together's inference endpoint with:

import { StreamToIterable } from '@axflow/models/shared';
import { TogetherAIInference } from '@axflow/models/togetherai/inference';

const stream = await TogetherAIInference.stream(
  {
    model: 'togethercomputer/llama-2-70b-chat',
    prompt: '[INST] Using no more than 20 words, what is the Eiffel tower? [/INST] ',
    max_tokens: 250,
  },
  {
    apiKey: process.env.TOGETHERAI_API_KEY,
  }
);

for await (const chunk of StreamToIterable(stream)) {
  console.log(chunk);
}

Support openAI tools:

• Multiple tools per API call
• Support streaming partial tools with new toolCallsAccessor callback

OpenAI made some changes to their SDK during their 2023 dev day, Axflow now supports them:

• Renaming of functions to tools
• Support for json_mode
• Support for seed and deterministic calls

https://txt.cohere.com/introducing-embed-v3/

Introduce a streamJsonResponse function that can pipe a ReadableStream through a Node.js ServerResponse object, enabling seamless integration for environments using the Node.js standard library, like Express.js.