Blog Post: Domain mode vs k8s/OpenShift blog post

[bstansberry]

(Caveat: this is a complex task likely to involve significant effort from the assignee and reviewers. Before starting on this be sure to talk to the SMEs in the area -- Brian Stansberry, Yeray Borges, Jeff Mesnil -- to confirm that they will be able to assist you.)

Domain mode is a long-standing WildFly feature, but our general recommendation is that users use Kubernetes or OpenShift. But we should write down in a more formal way why.

A general reason is both are meant to address the same use cases, but the communities working on these problems k8s and OpenShift dwarfs the community working on domain mode, so as a general principle it's wise to adapt the more widely supported approach. Even within the WildFly project we put more effort into k8s/OpenShift integration than into domain mode.

But a more detailed discussion would be good. It doesn't need to prove how k8s/OS is better for all areas, but it would be good to talk about the key domain mode use cases and how best practices with k8s/OS can handle those.

• Configuration consistency
• Rolling updates (affecting deployments and the base server)
• etc

A general theme can also be the general benefits of containers; things like the ability to test a consistent image, not one that varies based on the test environment.

Possibly this could end up becoming the foundation for content in the standard WildFly docs. Or we start there and the wildfly.org content becomes a simpler blog pointing at the published docs.

[jmesnil]

Dumping some ideas about that topic...

Moving from domain mode to Kubernetes presents some challenges that we should also cover.

• Immutable configuration means that the users must redeploy their images anytime they want to change WildFly configuration unless that's a runtime change that should affect live servers (eg pool flush, log level). This strongly suggests to adopt a GitOps approach where configuration (at large, including JBoss modules, etc.) is stored in a Git repo and outputs a new container image.
• A single container image contains both the WildFly server and the user deployments. Updating an application also requires a new image in order to be redeployed. A layered approach with a base image containing the server (built by a GitOps repo) and additional images with a single layer for the deployments would help separate server vs deployment concerns.
• Domain mode allows to run multiple servers on the same host but that's not practically achievable with Kubernetes where a container should only contain 1 single server.
• The ability to configure WildFly using env vars is key to adapt it to the Kubernetes cluster it targets.
• We should clarify if and how much kubectl rollout maps with domain mode's rolling updates.
• One of the biggest challenge with Kubernetes is that it can (and will) kills nodes (due to elasticity and cluster consumption) and the server must be properly set up to handle recovery to avoid losing any states. Having fine-tuned healthiness probes (to avoid false positive), right-sized resources requirements and a proper stable storage for data is a must for Kubernetes deployment

[jmesnil]

• How do we handle rollouts that contains WildFly upgrades that may have breaking runtime changes (eg Jgroups may not be compatible). Kubernetes resource rollout may not be enough and we might have to consider some blue/green deployment with Ingress. Assuming that the DevOps team is able to infer if the container updates is impacting by such change (@pferraro @rhusar)