Real Time Detection and Tracking System on UAV

Problem Statement Summary

Build a real time car/people detection and tracking system on DJI drones. The solution should include:

The videos should be captured by cameras on Drones.

Software framework should be based on ROS.

Car/people should be detected by deep learning methods and indicated by bounding box.

Positions and orientation control should be updated in real time (10 Hz or faster)

System Overview

Required Data Set

COCO( Pedestrian )

Data set(coco) has been obtained by Microsoft. COCO is a large-scale object detection, segmentation, and captioning dataset.

KITTI( Car )

Left color images of object data set 12GB

Training labels of object data set 5MB

Object development kit 1MB

Approach

Combine DJI M100, DJI X3 cameras, DJI Manifold(TK1), NVIDIA TX1 , ROS-indigo to generate result. The software on TX1 is a revised version of Nvidia embedding computing demo. The software on Manifold is a revised version of DJI-Onboard-SDK-ROS 3.1.

TX1:

Use KITTI dataset to training a new detectnet caffe model.(NVIDIA TX1 and Digits) Solver type = Adam Learning rate = 0.0001 Batch size = 2 Batch Accumulation = 5 Epoch = 30 pretrained model = Google Net

detectNet.cpp

Use TensorRT to accelerate the caffe model. (NVIDIA deep vision runtime library)

detectNet.h tensorNet.cpp

Program Cuda kernal to process each image and its bounding box(Cuda C/C++ API)

detectnet-console.cpp cuda

This net will return the coordinates of objects bounding boxes.(NVIDIA deep vision runtime library )

detectNet.cpp

Use ROS to send message between camera, manifold, TX1. (ROS image_transport sensor_msg cv_bridge std_msgs)

detectNet.cpp

TK1(DJI Manifold)

Use DJI X3 camera for video capture(25fps,1280*720, YUV->RGB) and use ROS to send image.(Onboard-SDK-ROS 3.1 DJI_SDK_read_cam) nv_cam.cpp

Use DJI manifold to process KCF algorithm for tracking return bounding box of tracked pedstrain/car (University of Coimbra C++ Implementation of KCF Tracker + ROS ) runtracker.cpp

Calibrate cameras to acquire essential matrix. uavcontrol.cpp

Transfer 2D coordinates to 3D coordinates by essentianl matrix and foundamental matrix for drone control. uavcontrol.cpp

Use DJI flying control SDK and implement a Kalman filter to optimize the control of DJI M100. (DJI Onboard-SDK-ROS 3.6 demo_flight_control) uavcontrol.cpp