Multimodal Sensor Fusion with Differentiable Filters

Models, training, and eval scripts for our IROS 2020 conference paper:

This repository contains our train/eval scripts crossmodal/unimodal weighted fusion architectures, and task-specific models and infrastructure. Filtering interfaces, dataset utilities, and implementations of standard particle filters, EKFs, and UKFs have been factored into a standalone library (torchfilter).

Repository Overview

.
├── crossmodal
│   ├── base_models             - General implementations for crossmodal and
│   │                             unimodal weighted fusion models.
│   ├── door_models             - PF, EKF, and LSTM models for door tasks.
│   ├── push_models             - PF, EKF, and LSTM models for pushing tasks.
│   └── tasks                   - Task definitions & configuration.
│
└── scripts
    ├── bash_scripts            - Bash script helpers for training.
    ├── door_task               - Training & eval scripts for door tasks.
    │   └── data_collection     - Data collection scripts.
    └── push_task               - Training & eval scripts for pushing tasks.

Additional code:

• Surreal Robotics Suite: for
  Panda arm simulations in MuJoCo.
• fannypack: for experiment
  management.

Installation

Package and dependencies can be installed in Python >=3.7 with pip:

git clone https://github.com/brentyi/multimodalfilter.git
cd multimodalfilter
pip install -e .