x.infer

<a href="https://github.com/dnth/x.infer/issues/new?assignees=&labels=Feature+Request&projects=&template=feature_request.md" target="_blank" rel="noopener noreferrer">Feature Request</a>

<a href="https://github.com/dnth/x.infer/issues/new?assignees=&labels=bug&projects=&template=bug_report.md" target="_blank" rel="noopener noreferrer">Report Bug</a>

<a href="https://github.com/dnth/x.infer/discussions" target="_blank" rel="noopener noreferrer">Discussions</a>

<a href="https://dicksonneoh.com/" target="_blank" rel="noopener noreferrer">About</a>

Why x.infer?

If you'd like to run many models from different libraries without having to rewrite your inference code, x.infer is for you. It has a simple API and is easy to extend. Currently supports Transformers, Ultralytics, and TIMM.

Have a custom model? Create a class that implements the BaseModel interface and register it with x.infer. See Adding New Models for more details.

Key Features

• Unified Interface: Interact with different machine learning models through a single, consistent API.
• Modular Design: Integrate and swap out models without altering the core framework.
• Ease of Use: Simplifies model loading, input preprocessing, inference execution, and output postprocessing.
• Extensibility: Add support for new models and libraries with minimal code changes.

Quickstart

Here's a quick example demonstrating how to use x.infer with a Transformers model:

import xinfer

model = xinfer.create_model("vikhyatk/moondream2")

image = "https://raw.githubusercontent.com/vikhyat/moondream/main/assets/demo-1.jpg"
prompt = "Describe this image. "

model.infer(image, prompt)

>>> An animated character with long hair and a serious expression is eating a large burger at a table, with other characters in the background.

Get a list of models:

xinfer.list_models()

 Implementation  Model ID                                         Input --> Output    

 timm            eva02_large_patch14_448.mim_m38m_ft_in22k_in1k   image --> class     
 timm            eva02_large_patch14_448.mim_m38m_ft_in1k         image --> class     
 timm            eva02_large_patch14_448.mim_in22k_ft_in22k_in1k  image --> class     
 timm            eva02_large_patch14_448.mim_in22k_ft_in1k        image --> class     
 timm            eva02_base_patch14_448.mim_in22k_ft_in22k_in1k   image --> class     
 timm            eva02_base_patch14_448.mim_in22k_ft_in1k         image --> class     
 timm            eva02_small_patch14_336.mim_in22k_ft_in1k        image --> class     
 timm            eva02_tiny_patch14_336.mim_in22k_ft_in1k         image --> class     
 transformers    Salesforce/blip2-opt-6.7b-coco                   image-text --> text 
 transformers    Salesforce/blip2-flan-t5-xxl                     image-text --> text 
 transformers    Salesforce/blip2-opt-6.7b                        image-text --> text 
 transformers    Salesforce/blip2-opt-2.7b                        image-text --> text 
 transformers    vikhyatk/moondream2                              image-text --> text 
 ultralytics     yolov8x                                          image --> objects   
 ultralytics     yolov8m                                          image --> objects   
 ultralytics     yolov8l                                          image --> objects   
 ultralytics     yolov8s                                          image --> objects   
 ultralytics     yolov8n                                          image --> objects   
 ultralytics     yolov10x                                         image --> objects   
 ultralytics     yolov10m                                         image --> objects   
 ...             ...                                              ...                 
 ...             ...                                              ...                 

Installation

[!WARNING] You must have PyTorch installed to use x.infer.

To install the barebones x.infer (without any optional dependencies), run:

pip install xinfer

x.infer can be used with multiple optional libraries. You'll just need to install one or more of the following:

pip install "xinfer[transformers]"
pip install "xinfer[ultralytics]"
pip install "xinfer[timm]"

To install all libraries, run:

pip install "xinfer[all]"

To install from a local directory, run:

git clone https://github.com/dnth/x.infer.git
cd x.infer
pip install -e .

Usage

Supported Models

Transformers:

• BLIP2 Series

model = xinfer.create_model("Salesforce/blip2-opt-2.7b")

• Moondream2

model = xinfer.create_model("vikhyatk/moondream2")

[!NOTE] Wish to load an unlisted model? You can load any Vision2Seq model from Transformers by using the Vision2SeqModel class.

from xinfer.transformers import Vision2SeqModel

model = Vision2SeqModel("facebook/chameleon-7b")
model = xinfer.create_model(model)

TIMM:

• EVA02 Series

model = xinfer.create_model("eva02_small_patch14_336.mim_in22k_ft_in1k")

[!NOTE] Wish to load an unlisted model? You can load any model from TIMM by using the TIMMModel class.

from xinfer.timm import TimmModel

model = TimmModel("resnet18")
model = xinfer.create_model(model)

Ultralytics:

• YOLOv8 Series

model = xinfer.create_model("yolov8n")

• YOLOv10 Series

model = xinfer.create_model("yolov10x")

• YOLOv11 Series

model = xinfer.create_model("yolov11s")

[!NOTE] Wish to load an unlisted model? You can load any model from Ultralytics by using the UltralyticsModel class.

from xinfer.ultralytics import UltralyticsModel

model = UltralyticsModel("yolov5n6u")
model = xinfer.create_model(model)

Adding New Models

• Step 1: Create a new model class that implements the BaseModel interface.

• Step 2: Implement the required abstract methods load_model, infer, and infer_batch.

• Step 3: Decorate your class with the register_model decorator, specifying the model ID, implementation, and input/output.

For example:

@xinfer.register_model("my-model", "custom", ModelInputOutput.IMAGE_TEXT_TO_TEXT)
class MyModel(BaseModel):
    def load_model(self):
        # Load your model here
        pass

    def infer(self, image, prompt):
        # Run single inference 
        pass

    def infer_batch(self, images, prompts):
        # Run batch inference here
        pass