Where should data plane configuration live?

[mpstefan]

As we progress NKG beyond just the Gateway API, we need a place to put additional configuration features for the data plane. Currently, our data plane is OSS NGINX. When we discussed it previously, we mentioned two distinct paths:

Configuration via NGINX ConfigMap

• Will require NGINX knowledge to configure
• May provide more functionality faster.
• May be used in tandem with NGINX template customization

Configuration via Custom Resource Definition

• Provides an abstraction layer from NGINX configuration
• Only exposes exactly what we intend
• Keeps user defined configuration in CRDs

We could also pursue a complete separate path. Here's some questions to prompt the discussion:

• How should we allow users to configure data plane specific features or configuration, such as log-level?
• How will the data plane configuration path decided on above work for future features?
• How will data plane specific feature configuration work with other potential NKG feature configuration?
• How difficult will it be for a user to find and implement the changes needed to make their change in the data plane?

[brianehlert]

Are you referring to "Global" settings that are generic to NGINX?
Such as logging, log customization, etc.

As opposed to snippets (context specific behavior injection) that would be context sensitive and thus I imagine extensions of the Gateway API CRDs.

[mpstefan]

At this point, I would argue yes - "Global" settings that apply to NGINX (i.e. nginx.conf) that have little value being unique per endpoint, hostname etc. This discussion was brought up originally because we were trying to determine an appropriate place to store log level configuration.

[brianehlert]

Then the question becomes: log level for what.
It is appropriate to want to turn up the log level for the control plane or for NGINX independently.

If the customer is troubleshooting traffic behavior, then they would want to turn up the logging at nginx.
If the customer is trying to figure out why their configuration is not being written to nginx.conf then they might turn up the control plane.

[pleshakov]

I think we need to have two compatible approaches:

• high-level approach, so the user can easily configure the most common configuration options
• low-level, so that the user can still configure NGINX, if the high level approach doesn't support some configuration option.

Below are some options for high-level and low-level control.

Note: we only addressing cluster operator persona, not application developers here -- https://gateway-api.sigs.k8s.io/concepts/security-model/#roles-and-personas

Approach	NGINX knowledge	Complexity for user (initial configuration + maintenance from upgrading an NKG release to the next version)	Level of control over NGINX	Something similar in NIC?
CRD with high level representation of NGINX configuration	Some knowledge	Easy	Modest	ConfigMap
ConfigMap with raw NGINX config	Good knowledge	Moderate	Extended	Snippets in the ConfigMap
ConfigMap with template modification (raw NGINX config + go template language )	Good knowledge	Hard	Almost full	Custom templates in the ConfigMap

[mpstefan]

The team met today and decided to move forward with this in an enhancement proposal for dynamic data plane configuration. In which, we'd like to put the "high-level" configuration in its own CRD and maintain the "low-level" configuration elsewhere, as pointed out in @pleshakov's post above. Furthermore, we decided that if we support additional types of data planes in the future, we will create a separate CRD for it.