Unity application for Microsoft HoloLens for use in ARCOR platform (augmented reality-based human-robot interaction). Application visualizes detected objects by robot ("to see what robot sees"), it provides visualization of robot learned programs which is fully controllable with voice. Furthemore, user is able to configure collision environment with use of gestures. The application communicates with ARCOR through a rosbridge, since the ARCOR runs under the Ubuntu 14.04 along with ROS Indigo.
I am using Unity 2017.4.7f1 version.
HoloLens needs to be calibrated due to the ARCOR system (world origin is in the table's right bottom corner). For calibration, Vuforia's image tracking was used. Markers can be found in ARCOR repo. When HoloLens detects marker, it draws virtual cube on it; when satisfied with detection, click on the cube to confirm it's position. Only three markers are used, which have to be placed as follows:
Robot
11--------------------------------(12)
| |
| |
10---------------------------------13
- Visualization of detected objects
- Visualization of learned robot programs
- Manipulation with collision objects – those, into which the robot is not allowed to enter (see video linked below)
- ROSBridgeLib – for communication with ROS
- Vuforia – for marker detection and calibration
- MixedRealityToolkit-Unity – version 2017.4.1.0
- Visualization of robotic programs using Microsoft HoloLens
- Collision Environment in Microsoft HoloLens
- Combining Spatial Augmented Reality Interface with Mixed Reality Head-mounted Display to Enhance User Understanding of Robotic Programs – paper presented on student conference Excel@FIT 2018
- User Interface for ARTable and Microsoft HoloLens – My Master's Thesis