Fiftyone viewer for SOLO Datasets
Introduction
pysolotools-fiftyone is a python package for viewing and interacting with solo datasets using the Voxel51 viewer.
This package allows the user to create a new Voxel51 viewer in either a web browser or jupyter notebook.
SOLO datasets are generated by Unity's Perception package. Please see here for more information about the SOLO schema.
It supports the displaying of the following annotations:
- Bounding Boxes
- 3D Bounding Boxes
- Keypoints
- Instance Segmentation
- Semantic Segmentation
- Depth
- Occlusion
- Normals
- Pixel Position
Pre-Requisites
- Install Anaconda or Miniconda (recommended).
- Install pre-commit.
- Install OpenEXR.
- Make sure
pipis installed
Installation
pip install pysolotools-fiftyone
Execution
CLI
The Solo Fiftyone viewer is executed on the command line with the following command line:
pysolotools-fiftyone examples/groceries
This will launch a new fiftyone viewer in your web browser.
Notebooks
SOLO FiftyOne can be executed in Jupyter Notebooks.
To use in a notebook:
from pysolotools_fiftyone.solo_fiftyone import run_in_notebook
path="./groceries"
run_in_notebook(path)
An example notebook has been included in the repository for your convenience.
The fiftyone will create an entry for each frame. Each entry will consist of a group with at least an RGB capture of the Unity game screen. If the dataset was created with Normals or Pixel Position labelers, the image group will also contain a PNG image for each one of them. These 3 types create the background of the frame, and the labels are drawn on to of them. The background image can be changed with the group dropdown located at the upper right of the screen.
Labels are drawn on top of the background group image. These can be toggled on and off via the Labels selector left of the image.
Annotation Types
Bounding Boxes
2D bounding boxes and label type for each visible labeled object in the image. Detailed information includes the amount of visible pixels of the object.
Semantic Segmentation
Semantic segmentation labels each pixel of an image with a label type.
Instance Segmentation
Each pixel of the image is labeled with the ID of the object that it represents.
Keypoints
Display keypoint (generally, but not limited to, a human skeleton) information.
3D Bounding Boxes
3D volume of object in the scene.
Depth
Pixel distance from the camera in the scene displayed in Plotly's virdis scale. The color scale goes from purple to yellow, with purple representing objects closer to the camera.
Occlusion
Object occlusion, or the amount of the object you can see, displayed in Plotly's virdis scale. The scale goes from yellow (fully visible) to purple (fully occluded). Hovering over a pixel will give you the exact occlusion value.
Normals
The normals image displays an object's surface normal in a 3 channel image. The red channel represents the normal's x value, the green channel represents the y channel, and the blue channel represents the normal's z value.
Pixel Position
The pixel position image displays a pixels cartesian coordinate in the camera's space. The red channel represents the pixel position's x value, the green channel represents the y channel, and the blue channel represents the z value. The image displayed is a 8bit per channel representation of a 32bit per channel file format, so some accuracy is lost in the display.
Sample SOLO Datasets
Inside the examples directory there are two sample datasets
- groceries: A dataset of grocery items that have all supported labels except keypoints
- keypoints: A simple dataset that has keypoint annotations
Community and Feedback
The Unity Computer Vision tools are open-source, and we encourage and welcome contributions. If you wish to contribute, be sure to review our contribution guidelines and code of conduct.
Support
For feature requests, bugs, or other issues, please file a GitHub issue using the provided templates we will investigate as soon as possible.
License
Know Issues
- When running with a web browser, it will open with a subset of your frames visible.
The data is still being imported in the background, and refreshing the browser will
update the viewer will the current set of loaded scenes. - To properly see some heatmap overlays (for instance depth and occlusion) make sure that you have Voxel51 settings (gear icon at the top of the screen) fields Color by set to label and Label opacity to highest (all of the way to the right)
