In Affordance Learning for End-to-End Visuomotor Robot Control, we introduced a modular deep neural network structure, that detects a container on a table, and inserts a ball into it.
We showed that our system performs its task successfully in zero-shot sim-to-real transfer manner. Each part of our system was fully trained either with synthetic data or in a simulation. The system was invariant to, e.g., distractor objects and textures.
The system structure:
We have didvided our work into the following code blocks:
- AffordanceVAED extracts affordance information from an observation image, and represents it as a latent space vector. Figure 1 shows the structure of the model.
- BlenderDomainRandomizer generates a domain randomized dataset for VAED.
- TrajectoryVAE represents trajectories in a low-dimensional latent space, and generates a trajectory based on a given latent vector.
- affordance_gym generates training data for TrajectoryVAE, and combines VAED and TrajectoryVAE together to perform desired trajectories based on an observation.
Ros Kinetic, MoveIt!, and MuJoco (2.0) should be installed. Colcon was used to build a workspace.
Prerequisites:
pip install -r affordance_gym/requirements.txt
sudo apt install ros-kinetic-moveit ros-kinetic-libfranka ros-kinetic-franka-ros
The work environemnt depends also on the
Workspace creation:
mkdir -p ~/ros/src
cd ~/ros/src
git --recursive clone git@github.com:gamleksi/affordance_gym.git
cd ~/ros
colcon build
The folder structure:
- Lumi Testbed by Intelligent Robotics group: The set of core ROS packages for Lumi, our robot. Contains URDF description, moveit configuration, mujoco configuration.
- Mujoco Ros Control by Intelligent Robotics group: Interface for the MuJoCo simulator.
- Lumi Pose Estimation: computes the camera pose of Kinect with an aruco marker, which is required in kinect_simulation.launch.
- Affordance Gym: Affordance Learning for End-to-End Visuomotor Robot Control
- Generate a domain randomized dataset for affordance detection, train a VAED model, and update
VAED_MODELS_PATH
in src/env_setup/env_setup.py to match to the model's parent folder path. - Generate trajectory training data run
python scripts/generate_perception_data.py
, train a TrajectoryVAE model, and update TRAJ_MODELS_PATH (in src/env_setup/env_setup.py) to match to the model's parent folder path. - Update
POLICY_MODELS_PATH
(in src/env_setup/env_setup.py) to match to the model's parent folder path and runpython scripts/generate_perception_data.py
to generate training policy data. - To train a policy model run
python scripts/perception_policy_train.py
, and to evaluate the policy model's performance in MuJoCo runpython scripts/perception_policy_train.py
. - Update
KINECT_EXPERIMENTS_PATH
(in src/env_setup/env_setup.py) and runpython scripts/kinect_env.py
to experiment the learned model with a real camera and with or without a real robot.
More info about each phase run python scripts/file_name.py -h
.
The table shows which launch file is required to be running with each python file.
Script | Simulation |
---|---|
generate_perception_data.py | roslaunch lumi_mujoco table_simulation.launch |
generate_trajectories.py | roslaunch lumi_mujoco table_simulation.launch |
kinect_debug.py | - |
kinect_env.py | roslaunch lumi_mujoco kinect_simulation.launch (without a real robot arm) |
kinect_test.py | roslaunch lumi_mujoco kinect_simulation.launch |
mc_interface.py | (real robot launch file) |
perception_policy_eval.py | roslaunch lumi_mujoco table_simulation.launch |
perception_policy_train.py | - |