This project was created as part of a Seminar - SupRTwin: Sensing, Understanding, and Provisioning of Robotic Digital Twins from Universität des Saarlandes. The objective of the project is to create a Digital Twin of objects that are detected during an inspection flight conducted by a drone. The Digital twins of the objects and the drone are represented in a realtime simulation environment in Unity Game Engine.
Original Repository: https://mrk40.dfki.de/drone-inspection-seminar/droneinspection
This repository contains the presentation files used for the final seminar presentation and some set of Experiment data.
The end to end flow looks like:
Have a look at the working Demo here:
The drone uses MQTT Protocol to communicate. In short, the following topics are of our interest
- Mavic2/state/pose - Pose information obtained from markers of drone using a Motion Capture system (OptiTrack system) and published to MQTT through Motive App
- Mavic2/state/physical - Current Physical State obtained using DJI SDK and Android dronecontrol app
The drone can be monitored live and inputs such as waypoints and certain commands such as starting live image can be provided to the drone through the Live Monitoring Web App.
- The project involves creation of two main services - Inspection Flight and Object Detection. They are created using Flask-RestX.
- We use custom trained YoloV5 model to detect the objects
- We use inverse Homography technique to estimate the pose of the objects detected.
- The flow of the pose estimation is given below
- The process control is done with the help of Camunda Process Model and PackML standard is used to maintain robotic states across the PPR Dashboard along with other robots.
- The states can be monitored in PPR Dashboard or using OPC UA Client.
- The PPR Dashboard is a part of the BaSys project which looks like the one below
- The digital twin of the detected object is then visualized using simulation framework - SiMRK
