Proposal for splitting a single container image into multiple container images

[youtalk]

The only final artifacts in the current Dockerfile are a devel image for the development container and a runtime image for the execution container.
https://github.com/autowarefoundation/autoware/blob/main/docker/autoware-openadk/Dockerfile

As a result, the size of container images can easily become bloated. In addition, modifying a single source code can cause the entire build to be redone, and the caching mechanism does not work effectively.

Therefore, we propose to split container images.
First, create an autoware-common image that contains only common message definition packages and common libraries, and run colcon build beforehand.
Inherit the autoware-common and create multiple container images using multi-stage build for each component of Autoware.
We are planning to configure the units of components according to the Autoware architecture diagram, such as autoware-perception for the Perception package, autoware-planning for the Planning package, and so on.

Multi-stage builds are expected to reduce the build time of container images and improve cache efficiency. Multiple containerization also makes it much easier to coordinate the computational resources of each Autoware component.
In addition, developers of Autoware components will only need to build the development containers for their own needs.
We believe this change will also greatly improve development and operational efficiency.

This concept is in line with the Open AD Kit, which promotes containerization. We look forward to your suggestions.

[hakuturu583]

I strongly agree with you.

I generally agree with the multi-stage build architecture, but I think there is a problem that modifying packages which was included in the autoware-common image or math libraries that may be used inside the various images will increase build time.

To improve the development experience, I think it is important for devel containers to cache apt/pypi packages and compile results using ccache, etc.

[youtalk]

Yes, thinner autoware-common stage becomes better development experience. I'm keeping in mind.

[doganulus]

Therefore, we propose to split container images.
First, create an autoware-common image that contains only common message definition packages and common libraries, and run colcon build beforehand.

@youtalk I have a similar view explained in this reply. This direction is excellent and applicable, but package developers and maintainers must address several points before containerization for successful implementation.

1. Containerization will need lists of packages for each container. A mechanism to tag packages by labels may be considered. The OpenADKit group cannot decide which package must go inside or not.
2. Containerization requires stricter dependency management by developers and maintainers. Adding a package dependency to bloat the whole container is too easy, as exemplified in my comment. Currently, 40% of the disk space in the runtime container is devel packages, which is unnecessary. But see that this is caused by rosdep install command, starting from the bloated rclcpp debian package. See this comment on the ROS forum regarding the container image bloat. The situation has stayed the same since then, as I see. I would appreciate any improvement on that front.
3. Furthermore, developers must always be encouraged and incentivized to reduce their dependencies to the absolute minimum, which is necessary for developing safety-critical systems. I would love to see a medium-term proposal from TierIV and everyone else regarding their dependency management strategies.

[youtalk]

@doganulus Thank you for sharing the very helpful information.
We will first do the development container splitting. At this point we still have one runtime container. The next step will be to split the runtime container and put it on an image size diet.
I will continue with the multi-containerization gradually and will keep you updated on the progress through pull requests.

[doganulus]

What would be the role of multiple containers in the workflow you think of?

Do you see Autoware developers develop their packages in their component devcontainer? Maybe I am wrong but my impression was that they mostly prefer native development. Yet, if happens, containerized development workflow would be a big achievement.

Runtime images are your end product. Therefore, you need to split runtime before splitting devel containers as developers need released runtime images of other components to develop in isolation. You will save a lot of time if you seriously apply runtime/devel package/container separation without delay. Please don't take the bad example of the ROS packaging practices. That's my number one worry in all that.

[youtalk]

I have already done large scale container development and operations with multiple isolated images on real consumer products as the person in charge.
Also, since Autoware is OSS, it is not practical to change everything out of the blue. Changes need to be made gradually. Separating the runtime container means that the execution method and documentation will all need to be changed. So we will start with the development container, which only affects developers.

[doganulus]

I think runtime images do not have any user base currently. Feel free to propose radical changes. This is a green space.

We want to use Autoware runtime and development container images but we are not able to do it effectively for various reasons, which often comes to the outdated ROS development habits. Containers are a chance to modernize all so we must use this chance well by questioning each and every feature. And discard all unnecessary or replace them modern equivalent. Your experience can be very welcome in all these efforts. I am glad you have joined!

[youtalk]

Since multi-stage build is used, it is more efficient to modify them in order from the upper stage. Therefore, we plan to maintain the development container first, followed by the runtime container.
We would like to expand container operations to make it easier for Autoware developers and users.

[oguzkaganozt]

Thank you for this great discussion @youtalk, I have created this recent discussion and PR for some initial modular containers (planning, simulator, visualizer), please feel free to review it everyone.