Drone pose estimation and Navigation with Unity

Note: This project has been developed with Python 3 and relies on Unity version 2020.3.7f1 and Unity Perception SDK 0.8.0-preview.2.

This project is on multi-object pose estimation and navigation, for a drone and a target landing pad. To do so, we used Unity’s perception package, in order to capture randomly generated synthetic data, which can be used to train a multi-object pose estimation model. This model can then be used to estimate the pose of our drone and target objects in a newly generated scene that was never seen before. The estimated position of the objects allows us to perform path planning, navigation, and obstacle avoidance, for landing the drone onto the target.

Want to skip the tutorial and run the full demo? Check out our Quick Demo.

Table of Contents

• Part 1: Create the Unity Project with Perception package
• Part 2: Setup the Unity Scene for Data Collection
• Part 3: Data Collection and Model Training
• Part 4: Inference and Navigation

Part 1: Create the Unity Project with Perception package

This part focuses on setting up the scene for data collection using the Unity Computer Vision Perception Package.We use the Perception Package Randomizers to randomize aspects of the scene in order to create variety in the training data.

Part 2: Setting up Unity scene for data collection

This part focuses on setting up the scene for data collection using the Unity Computer Vision Perception Package.We use the Perception Package Randomizers to randomize aspects of the scene in order to create variety in the training data.

Part 3: Data Collection and Model Training

This part includes running data collection with the Perception Package, and using that data to train a deep learning model. The training step can take some time. If you'd like, you can skip that step by using our pre-trained model.

If you want to have more information on how the model works and how to train your own, go here.

Part 4: Navigation and Inference

For inference the Unity scene connects to an external python process which that serves the trained model input/output. gRPC is a modern high performant Remote Procedure Call framework that can run in any environment. More information on the gPRC can be found here.

After getting the prediction we use Unity's Navigation and Pathfinding system that allows intelligent navigation of characters in the game world. This system uses navigation meshes that are automatically created from the scene geometry, called NavMesh.