CFK Schema Registry Basic Auth Vault Secret Auto-reload Failed

[nvolynets]

Hi,

Details

First of all, issue described below belongs more to CFK operator but unfortunately posting it here as failed to find corresponding escalation point for CFK.

Trying to deploy Schema Registry with enabled HTTP Basic Authentication on K8S via CFK operator with enabled Vault integration according to Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes.

All works fine as expected except basic auth credentials/ secrets rotation/ reload referred in jaas_config.conf file.

SchemaRegistry-Props {
  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
  file="/mnt/secrets/basic.txt"
  debug="true";
};

To apply updated credentials Schema Registry pod and/or stateful-set restart required.

In the same time there is as follows statement in official Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes doc:

Vault injects secrets into the CFK or Confluent Platform component pods on specific directory paths in the containers. Then, in the Confluent Platform component custom resources (CRs), the secrets are referenced via those directory paths (directoryPathInContainer).
If secrets are changed, Vault updates them in the pod, and CFK dynamically reads the updated info.

But it doesn't work as stated above.

STRs:

1. Enable HTTP Basic Authentication in Schema Registry as depicted in Confluent for Kubernetes API Reference (2.8) specification.
2. Deploy Schema Registry with Vault integration enabled according to Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes.
3. Successfully sign-in into Schema Registry (make some test request with using appropriate credentials specified in basic.txt Vault secret to verify Vault integration).
4. Update basic.txt Vault secret (just change password for some admin user).
5. Successfully check if corresponding updated secret has been delivered/ modified in corresponding Schema Registry pod.
6. Wait 10 minutes (next step will fail even if wait an hour).
7. Fail to sign-in into Schema Registry via using new creds (old still works).
8. Schema Registry pod and/or stateful-set restart required to reload basic.txt referred in jaas_config.conf file.

Having secrets exposed into Vault it is expected to rotate them from time-to-time. Unfortunately, forced each time restart Schema Registry is not an option here :(

Btw, for Kafka Broker & KsqlDB spinned up via CFK according to Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes. Secrets reload works as expected (automatically). It is yet another point to fix it for Schema Registry too to resurrect consistency.

Finally, the same issue applicable for Kafka Connect deployed via CFK but it looks like that it should escalated separately.

Analysis

Performed small investigation in this direction looks found out the root cause why updated secret/ creds are not reloaded automatically.

Confirmed that the secret automatically is updated in a corresponding Schema Registry pod by Vault Agent Injector operator.

But issue root cause looks like is in a jaas_config.conf file provided/ generated by CFK operator for Jetty app server: missed reloadInterval option. By default it is 0 which means "hot" props reload is disabled.
Providing below references to corresponding Jetty's classes source code:

• PropertyFileLoginModule.java#L106
• PropertyUserStore.java#L126

Consequently, as a proposal to sort it out please consider to extend CFK operator's Schema Registry CRD schema with providing possibility to specify reloadInterval option.