OpenBot

OpenBot leverages smartphones as brains for low-cost robots. We have designed a small electric vehicle that costs about $50 and serves as a robot body. Our software stack for Android smartphones supports advanced robotics workloads such as person following and real-time autonomous navigation.

MIT License

Stars
2.7K
Committers
35
View Code on GitHub Visit Website
Ecosystems: Arduino, Android
OpenBot - v0.7.1 Latest Release

Published by thias15 over 1 year ago

The following summarizes the changes compared to the previous release v0.7.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.7.0...v0.7.1

Robot App

Node.js Controller

Policy learning

OpenBot - v0.7.0

Published by thias15 over 1 year ago

The following summarizes the changes compared to the previous release v0.6.2.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.6.2...v0.7.0

Robot App

Android

iOS

Robot Body

Firmware

Documentation

OpenBot - v0.6.2

Published by thias15 over 1 year ago

The following summarizes the changes compared to the previous release v0.6.1.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.6.1...v0.6.2

Robot App

Policy training

Dependencies

OpenBot - v0.6.1

Published by thias15 almost 2 years ago

The following summarizes the changes compared to the previous release v0.6.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.6.0...v0.6.1

Robot App

Documentation & CI

Policy training

OpenBot - v0.6.0

Published by thias15 over 2 years ago

The following summarizes the changes compared to the previous release v0.5.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.5.0...v0.6.0

New Platforms

Android Apps

Robot App

Firmware

Documentation & CI

Policy Learning

OpenBot - v0.5.0

Published by thias15 over 2 years ago

The following summarizes the changes compared to the previous release v0.4.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.4.0...v0.5.0

Android Apps

Robot

Controller

NodeJS controller

Body

Firmware

Documentation

OpenBot - v0.4.0

Published by thias15 about 3 years ago

The following summarizes the changes compared to the previous release v0.3.2.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.3.2...v0.4.0

Android Apps

Robot

  • WebRTC support for communication with the smartphone controller with much lower latency, thanks @izivkov for #192
  • Add sensor selection menu to logger fragment, thanks @dhruv2295 for #202 and #228
  • Settings menu to choose between WebRTC (only controller app) and RTSP, thanks @dhruv2295 for #203
  • Fix app crash in object tracking fragment (Android 6), thanks @MohammedZ666 for #210 and #212
  • Dropdown to choose from available local web servers in logger fragment and autopilot fragment, thanks @sanyatuning for #214

Controller

  • WebRTC support and new features (mirror video feed and turn audio on/off), thanks @izivkov for #192
  • New option for intuitive robot control (throttle/brake with on-screen pedals and steering by tilting the phone), thanks @izivkov for #205

Web Interface

  • Add support for multiple connections between robots and local servers in the same network, thanks @sanyatuning for #214

Policy Training

  • Add weights & biases option to monitor training + some refactoring, thanks @sanyatuning for #214
  • Simplify image loading, thanks @dvdhfnr for #218
  • Create and use tf-records for training + unification of training code (notebook, terminal, web interface) + update to tensorflow version 2.5, thanks @marcelsan for #204

CI

  • Automated code style check for Python with black, #217
OpenBot - v0.3.2

Published by thias15 over 3 years ago

The following summarizes the changes compared to the previous release v0.3.1.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.3.1...v0.3.2

Android Apps

Robot

  • Split the AI screen into two separate screens, one for running object tracking and one for running autopilot models. Object detectors can now track all 80 object classes from the COCO dataset. Thanks @dhruv2295 for #179.
  • Refactor AI model definitions, #183
  • Refactor code to move common functions to util classes, #184
  • Improve stability of RTSP video stream, thanks @izivkov for #185
  • Bugfixes, thanks @dhruv2295 for #198
  • New model management screen that allows downloading more AI models, renaming/deleting model on phone and adding models from phone storage, thanks @dhruv for #188
  • Add permission handling to settings screen, thanks @dhruv for #199

The app has the following known issues:

  • Incorrect mapping of common BT game controllers (e.g. Xbox, PS4, Nvidia Shield, etc.) may occur on Android versions before Pie (9.0), see this article for more info. On the bright side, Android 11 supports many more controllers, see this article for more infos. Also, you can now use the button mapping screen to check what the app receives from your controller.

Controller

  • Add video streaming to controller app and python interface, thanks @izivkov for #182

Documentation

  • Update app documentation, #200
OpenBot - v0.3.1

Published by thias15 over 3 years ago

The following summarizes the changes compared to the previous release v0.3.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.3.0...v0.3.1

Android Apps

Robot

  • Add file picker to model drop-down --> choose trained autopilot models from phone storage, thanks @dhruv2295 for #172
  • Add Yolo-v4-tiny object detector, #174
  • Development of a new screen for checking the button mapping of the BT controller, thanks @dhruv2295 for #175
  • Fix navigation of AI fragment and add missing control handling, add audio effects for drive and speed modes, #180

The app has the following known issues:

  • Incorrect mapping of common BT game controllers (e.g. Xbox, PS4, Nvidia Shield, etc.) may occur on Android versions before Pie (9.0), see this article for more info. On the bright side, Android 11 supports many more controllers, see this article for more infos. Also, you can now use the button mapping screen to check what the app receives from your controller.

Controller

  • Reduce apk size of controller app, thanks @izivkov for #171
  • New keyboard controller --> control the robot from a Python script running on a computer connected to the same network, thanks @izivkov for #173

Documentation

  • Added QR codes for direct download of app apks, #178
OpenBot - v0.3.0

Published by thias15 over 3 years ago

The following summarizes the changes compared to the previous release v0.2.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.2.0...v0.3.0

Android Apps

Robot

  • Development of a new screen for free roaming and major refactor of the vehicle communication interface, thanks @dhruv2295 for #154
  • Development of a new screen for logging with ability to store preview and/or training data and upload to local server, thanks @dhruv2295 for #156
  • Development of a new screen to run autopilot and object detector models, thanks @dhruv2295 for #158
  • Phone controller bugfix, thanks @dhruv2295 for #159
  • Refactoring to improve USB connection and phone controller stability, thanks @dhruv2295 for #161
  • Add video streaming between robot and phone controller - WIP currently disabled due to latency, thanks @ivo for #162

The app has the following known issues:

  • Incorrect mapping of common BT game controllers (e.g. Xbox, PS4, Nvidia Shield, etc.) may occur on Android versions before Pie (9.0), see this article for more info. On the bright side, Android 11 supports many more controllers, see this article for more infos.

Controller

  • Refactoring of phone controller connection for added stability, thanks @izivkov for #157

You can download the apks from the resources below and install them on your phone directly. Note, that the app is signed only with a debug key. Alternatively, you can build the app yourself.

Documentation

  • Updated research paper (accepted for publication at ICRA'21)
  • Updated technical documentation for controller app, thanks @izivkov for #162
  • Documentation for new screens and instructions how to build new fragments, #161 and #170

Policy

  • Update requirements.txt, thanks @eagleanurag for #153
OpenBot - v0.2.0

Published by thias15 almost 4 years ago

The following summarizes the changes compared to the previous release v0.1.1.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.1.1...v0.2.0

Android Apps

Robot

  • Persistent app settings, thanks @francisduvivier for #126 and @sanyatuning for #127
  • Refactoring to make the app more modular and prepare for new GUI, thanks @dhruv2295 for #128, #134 and #141
  • Refactor vehicle interface, #129
  • Update UI to show vehicle stats, #130
  • Support for smartphone controller app, thanks @izivkov for #131
  • Send data to controller app, thanks @izivkov for #133
  • Automatically download standard ML models via Gradle, #135
  • Receive autopilot models from WebApp, thanks @sanyatuning for #144
  • Change speed modes from game controller, #146
  • Fix setting drive mode from UI, thanks @francisduvivier for #148

The app has the following known issues:

  • Incorrect mapping of common BT game controllers (e.g. Xbox, PS4, Nvidia Shield, etc.) may occur on Android versions before Pie (9.0), see this article for more info. On the bright side, Android 11 supports many more controllers, see this article for more infos.

Controller

  • New companion app that can be installed on another phone to control the OpenBot, thanks @izivkov for #137

You can download the apks from the resources below and install them on your phone directly. Note, that the app is signed only with a debug key. Alternatively, you can build the app yourself.

Documentation

  • Update Chinese docs, thanks @wangruoyao for #139
  • Add documentation for controller app, thanks @izivkov for #140
  • Add cool OpenBot projects, #145
  • Refactor documentation to separate robot and controller apps, #149

CAD Files

  • Fix STL files for glue body, #150

Policy

  • Restructure policy dir, create python package openbot, thanks @sanyatuning for #125
  • Add react based frontend for policy server, thanks @sanyatuning for #132
  • Download models from server, thanks @sanyatuning for #144
OpenBot - v0.1.1

Published by thias15 almost 4 years ago

The following summarizes the changes compared to the previous release v0.1.0.

Full Changelog: https://github.com/isl-org/OpenBot/compare/v0.1.0...v0.1.1

Documentation / CI

  • Added several images/screenshots and more detailed descriptions
  • Chinese translation for most of the docs, thanks @leegang for #84
  • Added shields with stats, thanks @eagleanurag for #81
  • Readme tweaks, thanks @custom-build-robots for #110 and @wangruoyao for #94
  • Automatically generate debug apks as artifacts and for releases, thanks @francisduvivier for #73, #74

Firmware

  • Major refactoring to make code more readable and compact
  • Removed NewPing library due to many reported issues (e.g. interference on pin 8, pin 3 and 11 not usable as PWM, etc.)
  • Use PinChangeInterrupt to allow for interrupts on all pins by parsing port interrupts
  • Implemented interrupt based ultrasonic sensor readings
  • Implemented interrupt based speed sensor readings for all hardware configurations
  • All main features/sensors can enabled/disabled with flags so the firmware only contains the functionalities needed and is as fast and extensible as possible
  • Improved NO_PHONE_MODE
  • Added OLED support to display vehicle state on the robot, thanks @custom-build-robots for #92

Android App

  • Added switch to toggle front/back camera, thanks @vkuehn for #115
  • Fixed #78: person following will now work correctly in any orientation of the phone
  • Added upload service for automatic log upload to http server, thanks @sanyatuning for #121

The app has the following known issues:

  • Some users with very old smartphones have reported incorrect mapping of the game controller
  • Only BT controllers detected as gamepad work (e.g. Xbox, PS4, Nvidia Shield, etc.)

You can download the apk from the resources below and install it on your phone directly. Note, that the app is signed only with a debug key. Alternatively, you can build the app yourself. If you want to deploy your own model, this is currently necessary.

Policy

  • Added python http server to receive logs from app and visualize them in browser, thanks @sanyatuning for #121
OpenBot - v0.1.0

Published by thias15 about 4 years ago

This is the first release. Several people were able to complete all parts of the project. The project is evolving and things may change a lot. The following summarizes current features and known issues for the different components.

Full Changelog: https://github.com/isl-org/OpenBot/commits/v0.1.0

3D-printed body

There are 3 different versions available, the regular body, the slim body and the glueable body. Depending on the build plate size of your 3D printer, at least one of those models should work. There are no known issues, but in the DIY version it is a bit challenging to fit all the wiring into the body.

Assembly

The repo includes instructions for a DIY build which is recommended for building a single robot to try out the project. In order to reduce the wiring and make assembly easier, a custom PCB is also available.

Custom PCB

There are two PCBs available, V1 and V2. Both versions of the PCB are supported by the Arduino firmware. However, there is no reason to order PCB V1 anymore, since PCB V2 has the same features and several (mostly minor) improvements:

  • Move the right speed sensor to pin D3 to enable interrupt functionality
  • Add Power LED for main battery
  • Update some components which are more commonly available
  • Update voltage divider to 20k/10k for better precision
  • Change motor connectors to upright version for easier access

Both PCBs have one known issue: When the ultrasonic sensor is used, the NewPing library causes an issue at pin 8. As a result, the right indicator light is very dim.

Firmware

The firmware has the following features:

  • support for three configurations: DIY, PCB_V1 and PCB_V2
  • receive control commands from Android application (USB serial)
  • produce low-level controls (PWM) for the vehicle
  • toggle left and right indicator signals
  • wheel odometry based on optical speed sensors
  • estimate battery voltage via voltage divider
  • estimate distance based on sonar sensor
  • send sensor readings to Android application (USB serial)

The firmware currently only supports the Arduino Nano microcontroller. However, porting it to other microcontrollers should be straight-forward. There are no known issues.

Android App

The app has the following features:

  • BT game controller interface for manual control of OpenBot
  • 3 different vehicle speeds (Slow - 50%, Normal - 75%, Fast - 100%)
  • Run neural network for person following
  • Run neural network for autonomous navigation
  • Data logger to collect datasets for training of autonomous driving policy

The app has the following known issues:

  • If the phone is mounted in the wrong orientation, the person following will drive in the opposite direction
  • On some phones (e.g. Asus Zenfone Max Pro (M2) ZB631KL) the front and rear camera are reversed (see issue #15)
  • Some users with very old smartphones have reported incorrect mapping of the game controller
  • Only BT controllers detected as gamepad work (e.g. Xbox, PS4, Nvidia Shield, etc.)

You can download the apk from the resources and install it on your phone directly. Note, that the app is signed only with a debug key. Alternatively, you can build the app yourself. If you want to deploy your own model, this is currently necessary.

The app was tested across 10 different smartphones from the past two years and works on all of them.

Jupyter Notebook

The notebook has the following features:

  • Data preprocessing and augmentation
  • Train neural networks for autonomous navigation (predefined models available)
  • Monitor training progress in tensorboard (live) and plot the evolution of several metrics (after training)
  • Visualize training data with labels and corresponding policy prediction
  • Convert the trained policy to a tflite model for deployment in the Android application

There are no known issues, but training a driving policy requires a lot of resources and may slow down or even freeze your machine. It is recommended to use a high-end laptop or workstation with large amount of RAM and dedicated GPU, especially when training with larger batch sizes. The documentation is currently also not very detailed.

What's Changed

Related Projects
gobot

Golang framework for robotics, drones, and the Internet of Things (IoT)

21 Sep 2013 8,714
machinascript-for-robots

Build LLM-powered robots in your garage with MachinaScript For Robots!

31 Jan 2024 158