S77_drone_summary.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nAvx4-UCNzt2"
   },
   "source": [
    "# Chapter Summary\n",
    "\n",
    "<img src=\"Figures7/S70-Autonomous_camera_drone-06.jpg\" alt=\"Splash image with ominous looking steampunk drone\" width=\"60%\" align=center style=\"vertical-align:middle;margin:10px 0px\">"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nAvx4-UCNzt2"
   },
   "source": [
    "In this final chapter, we upgraded to moving in 3D spaces, and explored an *under-actuated* autonomous system: a quad-rotor drone. Flying a drone needs some physics but this drone design is popular especially because the kinematics and dynamics are relatively simple. We discuss both advanced perception and planning methods that are frequently used in a drone context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "Drones move in 3D, so we resolutely moved to rigid transformations in three dimensions as describing the state of the drone, along with its linear and angular velocity. We introduced the $SE(3)$ manifold, which looks almost identical to $SE(2)$, but with a twist: rotations in 3D no longer commute. But from a practical point of view, rigid transformations are not much harder to deal with.\n",
    "\n",
    "The action space for drones is four-dimensional, corresponding to the four thrust vectors generated at the four rotors. Maintaining hover is fairly easy and gives us some ideas about how to dimension the motors, given a certain mass. We then explored forward flight and the relationship between tilt and forward velocity. Simulation of the drone's *kinematics* needed some interesting facts about 3D rotations, specifically the closed-form Rodrigues’ formula, which allows us to translate angular velocity into incremental rotations over time. Finally, *dynamics* was introduced to think about forces and torques, and the associated equations of motion.\n",
    "\n",
    "Compared to full rigid-body dynamics, we kept the exposition on sensing for drones relatively tame. We did introduce a very important new sensor: the inertial measurement unit or IMU. A typical IMU consist of a gyroscope, accelerometer, and magnetometer, and when used expertly, it can be an amazing sensor. However, there are tricky calibration and bias issues that make this one of the most difficult sensors to use correctly. Luckily, another very useful sensor for drones is the camera, or two cameras, in case we use them in a stereo configuration. A nice property of a stereo camera is that it allows us to measure points in 3D, but in a much more efficient way than a LIDAR.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning\n",
    "\n",
    "In the perception section, we explore *Visual SLAM* as an advancement of the \"SLAM with Landmarks\" by incorporating 3D poses, points.\n",
    "We lean into the use of cameras in unmanned aerial vehicles for their energy efficiency and compact size, which are crucial in energy-sensitive applications such as autonomous drones.  Additionally, we introduce methods for trajectory estimation using IMUs, and constructing 3D maps through Structure from Motion (SfM).\n",
    "Visual SLAM integrates these technologies in a real-time, incremental pipeline. \n",
    "While the implementation details of Visual SLAM are beyond the scope of this book, its potential in enabling autonomous flight through simultaneous inertial sensing and 3D reconstruction should be clear.\n",
    "\n",
    "In Section 7.5 we shift focus to trajectory planning for drones moving in 3D.\n",
    "To achieve optimal trajectories, which are critical for efficiency in time, energy, or distance, *trajectory optimization* is our key strategy. We could combine this with RRTs from earlier chapters to make discrete choices on how to act around obstacles. We start with the fundamental query of navigating from point A to B, illustrating the complexity introduced by factors like drone attitude and varying speeds at start and end points. \n",
    "We then expand upon the details needed for optimizing trajectories: defining the goal and  representing environments with cost maps that represent obstacles. Once again we use GTSAM and factor graphs for stating the resulting optimization problem and solving for it.\n",
    "After discussing time interpolation, we outline the implementation of a cascaded controller using drone kinematics and dynamics to execute these trajectories in simulation.\n",
    "\n",
    "Finally, in the final section of the book we focus on learning 3D representations from image data, specifically addressing the challenge of navigating new environments. Building on previous discussions about sparse Structure from Motion (SfM) and its limitations in providing dense representations for trajectory planning, we introduce a learning-based approach for creating detailed 3D scene models. We highlight Neural Radiance Fields (NeRF), a neural-based method for realistic scene rendering introduced in 2020, which supports motion planning and obstacle avoidance in drones. We examine the foundational concepts of NeRF and its application in robotics. We also discuss the possibility of integrating NeRF's density fields with motion planning techniques to enhance autonomous navigation in drones. However, note that neural scene representations are a fast-evolving field, and the details in this section might not stand the test of time, even if the intended application is expected to be served even better.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background and History\n",
    "\n",
    "The idea of using four rotors for vertical take-off and landing (VTOL) capabilities is even older than conventional helicopters, as evidenced by the full-scale contraption built by De Bothezat in 1921” ([George De Bothezat Helicopter on YouTube](http://www.youtube.com/watch?v=pjOcD5DoGUU)). This did not go anywhere, but of note is another attempt for a full-scale aircraft based on this principle called the X-22A by Bell ([X22-A on YouTube](http://www.youtube.com/watch?v=9CLxIebUejI)). The application of essentially the same principle for small unmanned aircraft can probably be attributed to the [1999 HoverBot project](https://ftp.eecs.umich.edu/people/johannb/paper99.pdf) at the University of Michigan, led by Johann Borenstein.\n",
    "\n",
    "A good contemporary overview on \"multirotor aerial vehicles\" is the review article by {cite:t}`Mahony12ram`.\n",
    "The kinematics and dynamics in 3D were inspired by the book {cite}`Murray94book` and the more recent and excellent \"Modern Robotics\" textbook by {cite:t}`Lynch17book_MR`.\n",
    "A more general treatment of small unmanned aircraft is {cite}`Beard11book`.\n",
    "\n",
    "An excellent text on inertial sensing and navigation is {cite}`Farrell08book`. The pre-integration of IMU measurements into binary factors was introduced by {cite:t}`Forster16tro`, inspired by an earlier paper by {cite:t}`Lupton12tro`. Structure from Motion aka 3D reconstruction from images is an old topic in computer vision, and for a deep dive two books come highly recommended: {cite}`Hartley00` and {cite}`Ma04book`.\n",
    "\n",
    "As mentioned, the controller in Section 7.5 was inspired by {cite}`Gamagedara19acc_geometric_control`. The idea of combining a controller with Neural Radiance Fields (NeRF) was presented in a 2022 Stanford paper by {cite:t}`Adamkiewicz22ral_nerf_nav`, and Section 7.6 on NeRF drew mostly on the original paper by Mildenhall et al. {cite:p}`Mildenhall22eccv_Nerf` as well as the later, voxel-based variant in the Direct Voxel Grid Optimization (DVGO) paper by {cite:t}`Sun22cvpr_dvgo`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "S11_sorter_state.ipynb",
   "provenance": []
  },
  "interpreter": {
   "hash": "c6e4e9f98eb68ad3b7c296f83d20e6de614cb42e90992a65aa266555a3137d0d"
  },
  "kernelspec": {
   "display_name": "Python 3.9.6 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "latex_metadata": {
   "affiliation": "Georgia Institute of Technology",
   "author": "Frank Dellaert and Seth Hutchinson",
   "title": "Introduction to Robotics"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}