Implementing BEVDet in Autoware

[cyn-liu]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I've agreed with the maintainers that I can plan this task.

Description

BEVDet is a BEV perception algorithm based on panoramic cameras. It unifies multi-view images into the perspective of BEV for 3D object detection task. It is different from the current 3D perception feature of Autoware.
BEVDet code repos

Purpose

Integrating BEVDet into Autoware for 3D object detection based on multi-view images, this task related to Sensing& Perception task.

Possible approaches

BEVDet is a 3D object detection model trained on NuScenes dataset using 6 surround view camera images. The 6 cameras form a 360 degree field of view with overlapping fields of view. When mapping from 2D to 3D, some parameters are required, including camera intrinsic parameters and extrinsic parameters between each camera and ego.
Integrating BEVDet into Autoware involves the placement of 6 cameras and calibration. Convert BEVDet model into ONNX format for deployment in Autoware.

Definition of done

• The placement of 6 cameras and calibration
• Convert BEVDet model into ONNX format
• Deploying BEVDet model on device using TensorRT
• BEVDet output result adaptation to Autoware topics

[liuXinGangChina]

Great，maybe you can make a todo task list first and see what others can take part in

[cyn-liu]

We refer to this project and successfully ran it on our own machine.
We use RTX3080 GPU and Trt FP16 inference BEVDet-R50-4DLongterm-Depth model. The mAP and inference speed of BEVDet-R50-4DLongterm-Depth TensorRT version can refer this project link.
The following is the running results on our machine:

v1.mp4

The following is the inference speed on our machine:

v2.mp4

Next, we will modify ROS1 node to ROS2 node based on this project, then, we will use TIER IV's dataset for testing, and we hope that this dataset can provide ROS2 bag format.

Our plan of integrate the BEVDet ROS2 node into Autoware:

1. define a bevdet_node in Autoware perception module
2. organize the 3D boxes results into autoware_perception_msgs::msg::DetectedObjects type
3. input the output result of bevdet_node into the object_merger node and fuse it with the detection results of other models