- [09/11/2023] MetaGraspNetV2 Online: MetaGraspNetV2 paper (Open Access) is published.
- [09/11/2023] Amodal Segmentation Masks for Real-World Data: MetaGraspNetV2 provides amodal masks and object poses for real-world data.
- [04/11/2022] Physics Simulation Online: Example code for simulating paralleljaw grasps in Isaac Gym available.
- [27/10/2022] Grasp Sampling Online: Example code for sampling of paralleljaw and vacuum grasp labels available.
- [26/07/2022] Additional Synthetic Data: Additional training data available.
Create custom conda environment and activate it.
conda env create -f environment.yml
conda activate metagraspnet_env
Prerequisites : Miniconda3, Ubuntu 20.04, Python 3.8. GPU not needed for working with dataset!
You can download the synthetic dataset (MGN-Sim) from Link. Unzip and save in ./dataset_sim.
You can download the real world dataset (MGN-Real) set from Link. Unzip and save in ./dataset_real.
Please Note: We recommend working with MetaGraspNetV2 data (see above).
You can download the synthetic dataset from Link. Unzip and save in ./dataset_sim.
You can download the real world dataset set from Link. Unzip and save in ./dataset_real.
Please consider citing the comprehensive journal version.
MetaGraspNetV2 - Journal Version Open Access (Link to Paper)
@ARTICLE{metagraspnetv2,
author={Gilles, Maximilian and Chen, Yuhao and Zeng, Emily Zhixuan and Wu, Yifan and Furmans, Kai and Wong, Alexander and Rayyes, Rania},
journal={IEEE Transactions on Automation Science and Engineering},
title={MetaGraspNetV2: All-in-One Dataset Enabling Fast and Reliable Robotic Bin Picking via Object Relationship Reasoning and Dexterous Grasping},
year={2023},
volume={},
number={},
pages={1-19},
doi={10.1109/TASE.2023.3328964}}
MetaGraspNet - Conference Version (Link to Paper)
@INPROCEEDINGS{metagraspnet,
title = {{MetaGraspNet}: A Large-Scale Benchmark Dataset for Scene-Aware
Ambidextrous Bin Picking via Physics-based Metaverse Synthesis},
author = {Maximilian, Gilles and Chen, Yuhao and Winter, Tim Robin and Zeng, E. Zhixuan and Wong, Alexander},
year = {2022},
booktitle = {IEEE International Conference on Automation Science and Engineering (CASE)}
}
You can visualize scenes in 2D and 3D. For visualizing the real world data, add --real_data, --real_data_grasps, or --real_data_amodal (MetaGraspNetV2) argument in 2D.
python Scripts/visualize_2d.py --data_root ./dataset_real --viewpt 3 --scene 418 --real_data --real_data_grasps --visualize_layout --real_data_amodal
python Scripts/visualize_2d.py --data_root ./dataset_sim --scene 1557 --viewpt 0 --visualize_layout
python ./Scripts/colored_pcl.py --data_root ./dataset_real --scene 418 --viewpt 3
python ./Scripts/colored_pcl.py --data_root ./dataset_sim --scene 1557 --visualize_parallel_gripper --colorize_per_score analytical --viewpt 0
You can download the models from Link. A config file of the labels is here.
We advise to use meshlab for viewing the .obj meshes.
The groundtruth for each viewpoint is contained in .../scene*/*_scene.hdf5 file. You can read it with h5ls.
h5ls ./dataset_sim/scene0/0_scene.hdf5
The file is structured as follows
|>camera
|>poses_relative_to_world
|>keypts
|>com
|>keypts_relative_to_camera
|>object_id
|>byhand
|>keypts_relative_to_camera
|>object_id
|>occluded_keypts_relative_to_camera
|>occluded_object_id
|>non_colliding_grasps
|>paralleljaw
|>contact_poses_relative_to_camera
|>contact_width
|>franka_poses_relative_to_camera
|>object_id
|>score_analytical
|>score_simulation
|>score_wrench
|>suctioncup
|>contact_poses_relative_to_camera
|>object_id
|>score_analytical
|>score_simulation
|>suction_poses_relative_to_camera
|>objects
|>bbox_loose
|>categories
|>poses_relative_to_camera
h5ls ./models_ifl/064/textured.obj.h5ls
The file is structured as follows
|>grasps
|>paralleljaw
|>pregrasp_transform
|>quality_score
|>quality_score_simulation
|>suctioncup
|>pregrasp_transform
|>quality_score
|>keypts
|>byhand
|>com
You can explore the individuals objects labels with:
Parallel-Jaw Grasp Labels
python ./Scripts/visualize_labels.py --root ./models --dataset_name models_ifl --object 064 --parallel_grasps --simulation --score_min 0.8 --max_grasps 100
Vacuum Grasp Labels
python ./Scripts/visualize_labels.py --root ./models --dataset_name models_ifl --object 064 --suction_grasps --analytical --max_grasps 500
Keypoints
python ./Scripts/visualize_labels.py --root ./models --dataset_name models_ifl --object 008 --keypts_byhand
Center of Mass
python ./Scripts/visualize_labels.py --root ./models --dataset_name models_ifl --object 008 --keypts_com
Our proposed dataset is already very comprehensive, however metaverses allow for customizing data generation. We provide scripts for that as well, you can find them below.
You can add custom .obj meshes by following the provided file structure in ./models.
We provide scripts to generate your own vacuum grasps based on our proposed cup model:
python ./grasps_sampling/scripts/sample_grasps.py --mesh_root ../models/models_ifl/ --suction --max_grasps 10
You can sample antipodal grasps with:
python ./grasps_sampling/scripts/sample_grasps.py --mesh_root ../models/models_ifl/ --paralleljaw --max_grasps 10
For generating parallel grasps based on physics simulation, please fullfill installation process from IsaacGym. Scripts are tested for isaac gym version 1.0.preview2. After you have set up a working isaac gym environment, start simulating with: (existing simulation data will be overwritten!)
(rlgpu) python ./physics_simulation/paralleljaw_simulation.py --root ../models/models_ifl/ --visualize --num_envs 16 --categories 008
We provide scripts which enable you to create your own custom dataset. After successful installation of Isaac Sim Python API, you can start creating custom data with
(isaac-sim) python ./scripts/dataset_physics.py --categories power_drill --root /home/XYZ/metagraspnet/models --dataset_name models_ifl [--randomize]
The dataset is under CC BY-NC-SA 3.0 license. You may only use the database for academic purposes. Any kind of commercial or military usage forbidden. For more details see.
MetaGraspNet_v0: Early preprint (2021) of MetaGraspNet project here. We do not recommend to cite this version!
MetaGraspNet: Conference version (IEEE CASE 2022) of our MetaGraspNet project here.
MetaGraspNetV2: Comprehensive journal version (IEEE T-ASE 2023) of our MetaGraspNet project here.