3.0.2 (2024-04-25)

Make

• Change NPM name (e97977e)

3.0.1 (2024-04-25)

Make

• Upgrade CI (83fc71a)

Multimodal Embeddings for JavaScript, Swift, and Python

How many AI models can run on-device out of the box? UForm multimodal embeddings can 🥳

JavaScript

Load the models and preprocessors for different modalities:

import { getModel, Modality, TextProcessor, TextEncoder, ImageEncoder, ImageProcessor } from '@unum-cloud/uform';

const { configPath, modalityPaths, tokenizerPath } = await getModel({
    modelId: 'unum-cloud/uform3-image-text-english-small',
    modalities: [Modality.TextEncoder, Modality.ImageEncoder],
});

Embed images:

const imageProcessor = new ImageProcessor(configPath);
await imageProcessor.init();
const processedImages = await imageProcessor.process("path/to/image.png");

const imageEncoder = new ImageEncoder(modalityPaths.image_encoder, imageProcessor);
await imageEncoder.init();
const imageOutput = await imageEncoder.encode(processedImages);
assert(imageOutput.embeddings.dims.length === 2, "Output should be 2D");

Embed queries:

const textProcessor = new TextProcessor(configPath, tokenizerPath);
await textProcessor.init();
const processedTexts = await textProcessor.process("a small red panda in a zoo");

const textEncoder = new TextEncoder(modalityPaths.text_encoder, textProcessor);
await textEncoder.init();
const textOutput = await textEncoder.encode(processedTexts);
assert(textOutput.embeddings.dims.length === 2, "Output should be 2D");
await textEncoder.dispose();

Swift

Embed images:

let imageModel = try await ImageEncoder(modelName: "unum-cloud/uform3-image-text-english-small")
let imageURL = "https://github.com/ashvardanian/ashvardanian/blob/master/demos/bbq-on-beach.jpg?raw=true"
guard let url = URL(string: imageURL),
    let imageSource = CGImageSourceCreateWithURL(url as CFURL, nil),
    let cgImage = CGImageSourceCreateImageAtIndex(imageSource, 0, nil) {
    throw Exception("Could not load image from URL: \(imageURL)")
}

var imageEmbedding: Embedding = try imageModel.encode(cgImage)
var imageVector: [Float32] = embedding.asFloats()

Embed queries:

let textModel = try await TextEncoder(modelName: "unum-cloud/uform3-image-text-english-small")
let text = "A group of friends enjoy a barbecue on a sandy beach, with one person grilling over a large black grill, while the other sits nearby, laughing and enjoying the camaraderie."
let textEmbedding: Embedding = try textModel.encode(text)
let textVector: [Float32] = textEmbedding.asFloats()

Python

Load model:

from uform import get_model, Modality

model_name = 'unum-cloud/uform3-image-text-english-small'
modalities = [Modality.TEXT_ENCODER, Modality.IMAGE_ENCODER]
processors, models = get_model(model_name, modalities=modalities)

Embed images:

import requests
from io import BytesIO
from PIL import Image

image_url = 'https://media-cdn.tripadvisor.com/media/photo-s/1b/28/6b/53/lovely-armenia.jpg'
image = Image.open(BytesIO(requests.get(image_url).content))

processor_image = processors[Modality.IMAGE_ENCODER]
model_image = models[Modality.IMAGE_ENCODER]
image_data = processor_image(image)
image_features, image_embedding = model_image.encode(image_data, return_features=True)

Embed queries:

text = 'a cityscape bathed in the warm glow of the sun, with varied architecture and a towering, snow-capped mountain rising majestically in the background'

model_text = models[Modality.TEXT_ENCODER]
processor_text = processors[Modality.TEXT_ENCODER]

text_data = processor_text(text)
text_features, text_embedding = model_text.encode(text_data, return_features=True)

Thanks to @xenova and @sroussey for help with JavaScript!
Thanks to @vmanot and @pcuenca for their work on Swift!

2.1.1 (2024-04-16)

Fix

• Importing ViT in gen_model.py (#80) (21f49ba), closes #80

2.1.0 (2024-04-14)

Add

• Initial Swift support (00bd84c)

Fix

• Image preprocessing in Swift (f2772d0)

Improve

• Fetching nested configs (729b9d9)

Make

• Formatting Swift code (f6faf4c)

2.0.2 (2024-03-28)

Make

• Fix PyPi CI version with hash (364afe6)

2.0.1 (2024-03-28)

Make

• PyPi upload version (9453802)

Today we are releasing a new batch of multimodal models trained with Nebius and already available on HuggingFace 🤗

1. Matryoshka style multimodal embeddings ranging from 64 to 256 and 768 dimensions 🖼️
2. Improved multimodal chat in 1.2B parameters, tuned with Direct Preference Optimization 💬
3. ONNX backend, making PyTorch dependency optional for lightning fast deployments ⚡

Great thanks to @lmmx, @blackforestboi, and @kapulkin for their patches to the project!

• Performance observations for M2 CPUs (#56) (8374ef6), closes #56
• Passing labels to text_decoder to compute loss. (#65) (f445a8b), closes #65
• Larger batch benchmarks (fdc8587)
• pre-commit config and linters (#62) (0a3efac), closes #62

1.1.0 (2024-02-15)

Add

• gen2 model (#66) (37c26bc), closes #66

1.0.3 (2023-12-29)

Improve

• basic benchmark (042ae87)

1.0.2 (2023-12-28)

Make

• Deprecate Anaconda (1ec8097)

UForm v1: Multimodal Chat in 1.5 Billion Parameters

The UForm family of tiny multimodal transformer models just got bigger! In addition to the existing CLIP-like embedding models, we now have a generative model useful for image captioning, visual question answering, and multimodal chats. All that is in just a billion parameters, small enough to fit even on mobile devices 🎉

Repository: https://github.com/unum-cloud/uform
Generative model: https://huggingface.co/unum-cloud/uform-gen
Chat model: https://huggingface.co/unum-cloud/uform-gen-chat

Evaluation Metrics

Being the smallest model of its kind, unum-cloud/uform-gen is hard to compare to others. Next in size are the 5x larger LLaVAs and InstructBLIP, with 7 billion parameters. LLaVA performs noticeably better on VQAv2: 78.5 vs 66.5. On captioning, CLIPScore and RefCLIPScore are relatively close across all models.

Throughput

On RTX 3090, using vanilla PyTorch for inference, with bfloat16 arithmetic and greedy decoding, one should expect the following numbers for throughput.

0.4.8 (2023-10-13)

Make

• pass ANACONDA_API_TOKEN as env. var. (ed020d3)

0.4.7 (2023-10-13)

Make

• urllib3 after v2 breaks Anaconda pipeline (05ed238)

0.4.6 (2023-10-13)

Make

• depend on urllib3 (79f7519)

0.4.5 (2023-10-13)

Make

• Push to Anaconda (72a2de4)

0.4.4 (2023-09-20)

Docs

• Add "Training Objectives" (76bdba9)
• Shorter lines (87ddf43)
• Update speed table (5dfcfe8)

Improve

• Expose TextEncoder and other models classes (47d969b)

0.4.3 (2023-09-01)

Docs

• Improved intro (d52225c)
• Intro Accents (708ab38)
• New intro (a32cfb3)
• update citation information (766dd04)

Make

• Add rebase action (45c2976)

0.4.2 (2023-08-17)

Docs

• Update Graphcore code (baa18d5)

Fix

• Sphinx last version not work (c3a0cc7)