general_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align: center;font-size: 40pt\">Welcome!</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "<img src=\"../common/images/banner.jpeg\" width=100% />\n",
    "</p>\n",
    "\n",
    "This is, in theory, your entry gate to explain the general context for the learning activities that were built for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Description\n",
    "\n",
    "While building this lecture my goal was to give access to the most advance knowledge on 3D registration algorithms.\n",
    "Those algorithms are used around the world since almost 25 years, but very few university courses cover them fully.\n",
    "Thus, this lecture targets perception algorithms for autonomous vehicles (i.e., robots in general, not just cars), with a particular focus on point cloud processing streamed from lidar sensors.\n",
    "Themes covered are:\n",
    "- the special orthogonal group in 3D, the rotation group;\n",
    "- rigid transformations in 3D;\n",
    "- frame of references;\n",
    "- differential geometry;\n",
    "- the middleware Robot Operating System (ROS);\n",
    "- point cloud registration, including special topics on data associations, objective function, and outliers rejection;\n",
    "- when having access to the campus, field deployment of autonomous vehicles\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Context\n",
    "\n",
    "This course presents the implications of autonomous vehicles on society and some theoretical foundations of perception algorithms allowing autonomous navigation of a vehicle in three dimensions (3D). \n",
    "We will see the limitations and advantages of the different sensors necessary for autonomous navigation. \n",
    "The algorithms for registering 3D point clouds will be explained with particular attention to the various representations of rotations from Euler angles to Lie algebra. \n",
    "When taught in hybrid mode, the practical side of the course will use a robot equipped with a 3D lidar to collect field data and create a 3D map of an area of interest on campus.\n",
    "\n",
    "The knowledge acquired in this course is useful in many industrial areas such as automakers, large tech companies, game programmers, and georeferencing experts. \n",
    "For some time, the majority of automakers have been investing heavily in research and development groups to automate driving. \n",
    "One of the emerging solutions is the use of 3D sensors to locate vehicles in their environment. \n",
    "Also, the video game industry is actively looking for interactive ways to engage the player in virtual environments using 3D sensors. \n",
    "Finally, contactless inspection - whether of small objects, construction sites or machined parts - is essentially based on 3D sensors, which allow a detailed analysis of the physical characteristics of the surfaces.\n",
    "All these areas are based on common concepts of 3D reconstructions, which will be covered in this course.\n",
    "\n",
    "To fully appreciate the course, it is suggested:\n",
    "- to be functional on an operating system such as Ubuntu, \n",
    "- to have a light programming base in Python, or to be willing to invest a bit of time to transition from Matlab to Python, \n",
    "- to be able to use command line tools, and\n",
    "- to be at comfortable with vectors, linear algebra, and statistics. \n",
    "\n",
    "As the course is given in remote or hybrid mode, participants are expected to use their personal computer.\n",
    "You are expected to have:\n",
    "- a computer with a minimum of 16 GB of rams and 20 GB of free disk space,\n",
    "- a webcam and a microphone for video conferencing\n",
    "- an Internet connection fast enough for video conferencing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Who are the instructors?\n",
    "\n",
    "- Professor: [François Pomerleau](https://norlab.ulaval.ca/people/f_pomerleau/)\n",
    "- Instructor: [William Dubois](https://norlab.ulaval.ca/people/)\n",
    "- Instructor: [Damien LaRocque](https://norlab.ulaval.ca/people/)\n",
    "\n",
    "\n",
    "<p style=\"text-align:center\">\n",
    "<img src=\"../common/images/stay_focused.jpg\" width=75% />\n",
    "<br>\n",
    "Stay focused!\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning objectives\n",
    "\n",
    "If we try to picture the skills I would like you to have after this lecture, they could be listed as:\n",
    "  \n",
    "1. Understand the mathematical foundations for processing point clouds:\n",
    "   1. Understand the basics of differential geometry to extract structural information from a point cloud;\n",
    "   1. Manipulate 3D points by making landmark changes;\n",
    "   1. Convert between different 3D rotation formalism; and\n",
    "   1. Explain the advantages and disadvantages of the different rotation formalism.\n",
    "<br><br>\n",
    "1. Implement a point cloud registration algorithm:\n",
    "   1. Understand the basic operation of a lidar;\n",
    "   1. Compute a rigid transformation minimizing the alignment error between two point clouds;\n",
    "   1. Schematize a complete point cloud registration solution; and\n",
    "   1. Criticize a complete point cloud registration solution.\n",
    "<br><br>\n",
    "1. Design an autonomous vehicle perception system:\n",
    "   1. Understanding the social and technical implications of an autonomous vehicle;\n",
    "   1. Use middleware for data processing; and\n",
    "   1. Process a data stream from a lidar.\n",
    "<br><br>\n",
    "1. Plan time for continuous learning:\n",
    "   1. Autonomously fetching information in the class material;\n",
    "   1. Applying the acquired knowledge in a set of exercises; and\n",
    "   1. Planning your own deadlines and respecting them to reach the class objectives on time.\n",
    "\n",
    "<p style=\"text-align:center\">\n",
    "<img src=\"../common/images/campus3D.jpg\" width=75% />\n",
    "<br>\n",
    "Reconstruction of the campus using point clouds from lidar.\n",
    "The red line corresponds to the path driven by the robot, while the point cloud is colored based on elevation.\n",
    "During this lecture, you will process similar data to build 3D maps.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Teaching methods\n",
    "\n",
    "In that lecture, I'm experimenting different teaching methods proposed by the pedagogic supporting staff.\n",
    "A lot are uncommon teaching methods, so **be ready to experiment with me**.\n",
    "Your inputs are necessary to evolve this lecture in the direction that is the most useful for **you**, so don't hesitate to suggest ideas or provide recommendations for next year.\n",
    "\n",
    "The main method put forward is the [flipped classroom](https://en.wikipedia.org/wiki/Flipped_classroom#Limitations_and_criticisms) combined with asynchronous teaching.\n",
    "To implement this method, you become responsible for your learning pace and most try the activities first.\n",
    "By reading and trying the exercises as best you can, you will be able to better identify gaps in your understanding.\n",
    "This process can be uneasy or uncomfortable for some at the beginning as it may highlight learning challenges that you are facing.\n",
    "The goal of the instructor is to support you in your learning process and answer your questions to help you connect concepts.\n",
    "This approach favour active learning, as you are responsible to search for solutions, as opposed to passive learning where you receive a stream of information in one block and try to figure out how it will be useful.\n",
    "\n",
    "There will be one video at the beginning of each module to explain the general goals of that module.\n",
    "Following your demands, there will be videos explaining concepts that triggered a lot of questions or that cannot be explained with few text lines in a chat.\n",
    "To support fast, but still asynchronous communications, we will use a dedicated instant messaging app for the lecture.\n",
    "Instruction on how to access the app will be given at the beginning of the semester.\n",
    "You are encouraged to use public channels as much as you can so everyone can join the conversation.\n",
    "If you feel the need, you can also ask questions to instructors privately.\n",
    "    \n",
    "The following [xkcd comic strip](https://xkcd.com/1319/) resumes pretty well how I built this class, with the small note that I prefer coding than doing PowerPoints and talking three hours straight in front of a class...\n",
    "\n",
    "<p style=\"text-align:center\">\n",
    "<img src=\"../common/images/automation_2x.png\" width=50% />\n",
    "<br>\n",
    "To automate or not?\n",
    "After all, I'm doing robotics so why not?\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diversification of the teaching material\n",
    "\n",
    "The concept of a classroom where you need to sit, or watch, for a dedicated period of time will not be used for this lecture.\n",
    "Instead, we provide a variety of learning activities that should suit a larger spectrum of learning abilities.\n",
    "We all learn differently, so hopefully, by diversifying the teaching material, we will keep most of you interested in the lecture.\n",
    "Here is the list:\n",
    "- **Jupyter notebooks** will be used for interactive lessons. \n",
    "This material is where we put most of our effort to get this lecture interesting.\n",
    "A server hosting Jupyter hub is used to provide an interactive environment, which is mostly useful for lessons heavy on the theoretical aspects of the lecture.\n",
    "Instead of having access to only equations and PDF, we provide as much as we can implementations of  equations in Python for you to test and break.\n",
    "Don't fear to change the code and explore solutions.\n",
    "If you break your notebook, we provide a [public GitHub repository, `percep3d_students`,](https://github.com/norlab-ulaval/percep3d_students) for you.\n",
    "You can always go back and see what the original notebook looked like.\n",
    "As for the text, you can also see how the equations were written and take notes directly in notebooks.\n",
    "Don't hesistate to use GitHub issues to report any typos or error.\n",
    "- **Instant messaging** will be used for general information and to answers quick questions.\n",
    "I value quick feedbacks and would like to promote interactions instead of a unidirectional flow of information.\n",
    "- **Video conference** will be used for one-on-one communications.\n",
    "We will also use video conference for oral exams.\n",
    "Specific links to join a call will be sent through the instant messaging app.\n",
    "- **Videos** will be produced for guidelines, focus explanations and tutorials.\n",
    "We are using unlisted videos hosted on our [Norlab's YouTube channel](https://www.youtube.com/c/norlab), so you will receive direct links either directly in the notebooks or in Slack\n",
    "- **Reading** material are also provided as PDF files.\n",
    "You will be guided through them section by sections.\n",
    "- **Hands-on robotics tools** will be used to give you a better experience of what needs to be done to setup a robot.\n",
    "A virtual machine will be provided with all the necessary installation to save times.\n",
    "- **Training on field robotics** will be done on campus when the situation allows it.\n",
    "This activity will help you to better understand the complexity of programming physical devices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessibility\n",
    "\n",
    "I'm doing my best to provide the most accessible lecture I can with the resources I have.\n",
    "My personal goal is that you should not need any special arrangement to follow the lessons, as they were built directly for a broader student audience.\n",
    "The lecture aims at supporting both French and English. \n",
    "The written material is in English as most of the manuals are.\n",
    "For now, videos will be in French and I will eventually figure out how to support both languages and use closed caption.\n",
    "Of course, evaluations and answer to your questions will be in the language of **your choice**.\n",
    "\n",
    "I'm am also aiming at a worldwide audience with this lecture.\n",
    "We are using a [Creative Commons license](https://creativecommons.org/) for the lessons, so you can adapt and reuse as much as you can.\n",
    "All lessons are freely available to all at https://github.com/norlab-ulaval/percep3d_lessons.\n",
    "What you get from registering officially is support from specialized instructors, answers to exercises, grading and certifications (i.e., credit for the lecture counting in your training program).\n",
    "Otherwise, knowledge should be free.\n",
    "We also promote open source projects by using them instead of commercial solutions.\n",
    "Namely we use Jupyterlab, notebooks, nbgrader, Python, and ROS.\n",
    "Finally, reading materials were selected to be freely and openly available:\n",
    "  - Pomerleau, F., Colas, F., & Siegwart, R. (2015). **A Review of Point Cloud Registration Algorithms for Mobile Robotics**. _Foundations and Trends in Robotics_, 4(1), 1–104. [PDF provided by HAL](https://hal.archives-ouvertes.fr/hal-01178661/document)\n",
    "  - Barfoot T D. **State Estimation for Robotics**. Cambridge University Press, 2017. ISBN 9781107159396. doi: 10.1017/9781316671528. [PDF nicely provided by the author](http://asrl.utias.utoronto.ca/~tdb/bib/barfoot_ser17.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation methods\n",
    "\n",
    "Another goal is to diversify grading methods to suit different types of learning.\n",
    "To this end, we provide types of evaluation: exercises, assignments, and interviews.\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "    <th style=\"width:10%\"></th>\n",
    "    <th style=\"width:20%\">Exercises</th>\n",
    "    <th style=\"width:20%\">Assignments</th>\n",
    "    <th style=\"width:20%\">Special assignments</th>\n",
    "    <th style=\"width:20%\">Interviews & Exams</th>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Goal</td>\n",
    "    <td>Reverse teaching by asking you to find information to solve a given problem.</td>\n",
    "    <td>Ensure that you engage with the teaching material in a timely manner.</td>\n",
    "    <td>Value extra effort and engagement toward the teaching material.</td>\n",
    "    <td>Validate your understanding of different concepts and how you can put them into practice.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Process</td>\n",
    "    <td>You need to complete dedicated notebooks and upload them to <a href=\"https://monportail.ulaval.ca/\">MonPortail -> Évaluations et résultats -> Formatives</a>.\n",
    "You should help each other with the exercises as explaining concepts helps to stabilize knowledge.</td>\n",
    "    <td>You need to complete the dedicated notebook and upload it to <a href=\"https://monportail.ulaval.ca/\">MonPortail -> Évaluations et résultats -> Sommatives</a>.\n",
    "    You must do them individually and should reserve more time than typical exercises to complete them.</td>\n",
    "    <td>At a given lesson through the lecture, you will be proposed a special assignment that requires advanced knowledge and skills on a specific topic.\n",
    "Their grading is binary, either completed or not, and have the super power to replace your lower assignment grade with 100 %.</td>\n",
    "    <td>Examinations are done using videoconferencing tools and there is only one student with an instructor(s).\n",
    "Examinations are in a form of an interview, not a monologue from the student.\n",
    "The main difference with written exams is that the instructor is there to help you to progress through questions.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Constraints</td>\n",
    "    <td>They are mandatory but not graded individually as they are fully part of the lessons.\n",
    "    You will receive points if you submitted all the exercices on time at the end of the lecture.\n",
    "You should do them carefully as they will prepare you to the assignments and the interviews.\n",
    "If there is a general effort in trying the exercises, you will receive the answers in MonPortail.\n",
    "If not, you will encourage you to try again and ask more questions if you're blocked.\n",
    "</td>\n",
    "    <td>Refer to MonPortail and the lesson schedule to look for assignement deadlines.\n",
    "    Deadlines are set uniformly through the semester and follow the general lessons.</td>\n",
    "    <td>If you decide to do one of them, you need to warn an instructor before starting it.\n",
    "Those assignments are reserved for advanced students who already master a subtopic.</td>\n",
    "    <td>The duration is less than 20 min. \n",
    "The first five minutes are to give you instructions and time to prepare.\n",
    "You will need to fix an appointment in advance with an instructor to pass an exam.\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Percentage of the semester</td>\n",
    "    <td>5 %</td>\n",
    "    <td>25 %</td>\n",
    "    <td>from 0 % to 15 %</td>\n",
    "    <td>70 %</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Number of activities</td>\n",
    "    <td>many</td>\n",
    "    <td>5</td>\n",
    "    <td>from 0 to 3</td>\n",
    "    <td>3</td>\n",
    "</tr>\n",
    "</table>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Similar lectures\n",
    "\n",
    "The course content has an interdisciplinary scope aimed at students in computer science, software engineering, computer engineering, electrical engineering and mechanical engineering, and geomatics with interests in the following courses:\n",
    "\n",
    "| | |\n",
    "|:---------|:---------|\n",
    "|GLO-7021  |[Introduction à la robotique mobile](https://www.ulaval.ca/les-etudes/cours/repertoire/detailsCours/glo-7021-introduction-a-la-robotique-mobile.html) |\n",
    "|GMC-7046  |[Éléments de robotique](https://www.ulaval.ca/les-etudes/cours/repertoire/detailsCours/gmc-7046-elements-de-robotique.html)|\n",
    "|GIF-7001  |[Vision numérique](https://www.ulaval.ca/les-etudes/cours/repertoire/detailsCours/gif-7001-vision-numerique.html) |\n",
    "|GMT-7007  |[Capteurs actifs](https://www.ulaval.ca/les-etudes/cours/repertoire/detailsCours/gmt-7007-capteurs-actifs.html) |\n",
    "|GIF-7105  |[Photographie Algorithmique](https://www.ulaval.ca/les-etudes/cours/repertoire/detailsCours/gif-7105-photographie-algorithmique.html)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ... and in the world...\n",
    "\n",
    "After looking for a while and as far as I can tell, this lecture is the most advance training on point cloud registration you can get.\n",
    "That being said, some part of the lecture are generic, and we took inspiration from many other robotics lecture.\n",
    "Most of them offer free online material, so you could complement your training with them.\n",
    "\n",
    "\n",
    "[Programming for Robotics - ROS](https://rsl.ethz.ch/education-students/lectures/ros.html):\n",
    "- Place: ETH Zurich, Switzerland\n",
    "- Links with this lecture: Part of our training on ROS use their material.\n",
    "The interesting point of that lecture is that it is given in a mechanical department, so robot coding robots should not scare any of our none computer science students!\n",
    "\n",
    "\n",
    "[General training on ROS](http://wiki.ros.org/Courses):\n",
    "- Place: There are more than 33 universities in the world teaching ROS\n",
    "- Links with this lecture: Most of those lectures rely on the web-based tutorial provided by the Open Source Robotics Foundation.\n",
    "\n",
    "[Modern Robotics](https://www.coursera.org/specializations/modernrobotics):\n",
    "- Place: Northwestern University / Coursera, United States\n",
    "- Links with this lecture: The section on rigid body motions introduce similar concepts as what is shown in our modules \"Welcome to a flat world\" and \"How hard can it be in 3D?\".\n",
    "Moreover, they provide a [YouTube playlist](https://www.youtube.com/playlist?list=PLggLP4f-rq02vX0OQQ5vrCxbJrzamYDfx) of their lecture for free.\n",
    "\n",
    "[Self-Driving Cars Specialization](https://www.coursera.org/specializations/self-driving-cars)\n",
    "- Place: University of Toronto / Coursera, Canada\n",
    "- Links with this lecture: One course of this specialization is _State Estimation and Localization for Self-Driving Cars_ where some part cover lidars and point cloud registration.\n",
    "Also, the author of the book used for the more theoretical lessons is part of this teaching group.\n",
    "\n",
    "[Introduction to Mobile Robotics](http://ais.informatik.uni-freiburg.de/teaching/ws19/mapping/) and [Robot Mapping](http://ais.informatik.uni-freiburg.de/teaching/ss19/robotics/)\n",
    "- Place: University of Freiburg, Germany\n",
    "- Links with this lecture: The professor responsible for this lecture, Wolfram Burgard, is the father of SLAM (Simultaneous Localization and mapping).\n",
    "There is a good basis on lidars, rotations, but the teaching of point cloud registration remains in 2D.\n",
    "\n",
    "\n",
    "[Self-Driving Car Engineer Nanodegree Program](https://www.udacity.com/course/self-driving-car-engineer-nanodegree--nd013):\n",
    "- Place: Audacity, United States\n",
    "- Links with this lecture: One of Burgard's student, Sebastian Thrun, founded his own online university, so it is normal that a good part of the training focus on autonomous cars.\n",
    "More precisely, \"Module 4: Introduction to Localization\" proposes exercises on registration, but also remains in 2D.\n",
    "\n",
    "\n",
    "[Robotics: Estimation and Learning](https://www.coursera.org/lecture/robotics-learning/iterative-closest-point-1jDix)\n",
    "- Place: Pennsylvania University / Coursera, United States\n",
    "- Links with this lecture: That lecture offers an overview of 3D registration, but covers in 5 minutes what we will take two weeks to see in detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now it is your turn!\n",
    "\n",
    "Start here to access the [list of modules and their interconnections](./modules_overview.ipynb) for this lecture.\n",
    "\n",
    "<p style=\"text-align:center\">\n",
    "<img src=\"../common/images/wasnt_me.jpg\" width=100% />\n",
    "<br>\n",
    "It wasn't me!\n",
    "</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}