Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PROPOSAL] Support adding CSI volumes to operator to pull secrets from Key vault #795

Closed
nilushancosta opened this issue Apr 29, 2024 · 4 comments
Closed

[PROPOSAL] Support adding CSI volumes to operator to pull secrets from Key vault #795

nilushancosta opened this issue Apr 29, 2024 · 4 comments
Labels
untriaged Issues that have not yet been triaged

Comments

@nilushancosta
Copy link
Contributor

What/Why

What are you proposing?

Provide the capability to add CSI volumes to the operator pod inorder to pull secrets from Azure Key vault.

What users have asked for this feature?

What problems are you trying to solve?

At present, the admin credentials secret has to be created in the cluster before it can be used by the operator. But with this change, this secret can be added to the key vault and the CSI secrets store driver will get the secret from the key vault and create a secret automatically

What is the developer experience going to be?

No REST API changes

Are there any security considerations?

No

Are there any breaking changes to the API

No

What is the user experience going to be?

Users will have 2 new fields they can add to the operator's Helm values file. These can be used to add a CSI secrets store volume.

extraVolumes: []

extraVolumeMounts: []

Are there breaking changes to the User Experience?

No

Why should it be built? Any reason not to?

This will enable users to store the admin credentials secret in a keyvault and use it

What will it take to execute?

The operator's deployment manifest needs to be modified to add the volumeMounts field to the container spec and the volumes field to the pod spec
I can contribute this

Any remaining open questions?

No
@github-actions github-actions bot added the untriaged Issues that have not yet been triaged label Apr 29, 2024
@nilushancosta nilushancosta mentioned this issue May 6, 2024
Closed
6 tasks
@swoehrl-mw
Copy link
Collaborator

Hi @nilushancosta.
I'm not really sure how extra volumes for the operator pod can help with the admin credentials secret? That secret is per opensearch-cluster, so there are potentially any number of them per operator instance.
Could you please explain how you intend this to work? Maybe also give an example?

@nilushancosta
Copy link
Contributor Author

Hi @swoehrl-mw ,

I want to be able to use a CSI secrets store volume to create the adminCredentialsSecret and the securityConfigSecret used in the OpenSearchCluster manifest - https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#custom-admin-user

security:
  config:
    adminCredentialsSecret:
      name: admin-credentials-secret  # The secret with the admin credentials for the operator to use
    securityConfigSecret:
     name: securityconfig-secret  # The secret containing your customized securityconfig
  tls:
    transport:
      generate: true
    http:
      generate: true

I initially tried to use the CSI secret store driver through the additionalVolumes of the OpenSearchCluster. However the CSI secrets store driver will only create the secrets when the pod where the volumeMount is added starts (i.e. OpenSearch pod starts). But the operator will not start the OpenSearch pod unless the secret exists. It will keep printing the following logs

{"level":"info","ts":"2024-05-20T18:13:55.678Z","msg":"Waiting for secret 'securityconfig-secret' that contains the securityconfig to be created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"412104d5-6c66-4994-8aa5-2eed921b5130"}

{"level":"info","ts":"2024-05-20T18:14:27.079Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"05e35c83-31a1-4d67-b376-1633ceadcefc","interface":"transport"}
{"level":"info","ts":"2024-05-20T18:14:27.079Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"05e35c83-31a1-4d67-b376-1633ceadcefc","interface":"http"}
{"level":"error","ts":"2024-05-20T18:14:27.081Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"05e35c83-31a1-4d67-b376-1633ceadcefc","error":"Secret \"admin-credentials-secret\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

To solve this, I could add the following secret provider class and mount the CSI secrets store volume to the operator pod using the proposed fields as follows

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: opensearch
  namespace: test
spec:
  provider: azure
  secretObjects:
  - secretName: securityconfig-secret
    type: Opaque
    data:
    - key: internal_users.yml
      objectName: SECURITY_CONFIG_INTERNAL_USERS
  - secretName: admin-credentials-secret
    type: Opaque
    data:
    - key: username
      objectName: USERNAME
    - key: password
      objectName: PASSWORD
  parameters:
    keyvaultName: test-kv
    tenantId: aaa-bbb
    usePodIdentity: "false"
    objects: |
      array:
        - |
          objectName: internal-users
          objectType: secret
          objectAlias: SECURITY_CONFIG_INTERNAL_USERS
          objectVersion: 1111
        - |
          objectName: password
          objectType: secret
          objectAlias: PASSWORD
          objectVersion: 2222
        - |
          objectName: username
          objectType: secret
          objectAlias: USERNAME
          objectVersion: 3333
extraVolumeMounts:
- name: csi-volume-opensearch-secrets
   mountPath: "/mnt/csi-secret-store/opensearch-secrets"
   readOnly: true
extraVolumes:
- name: csi-volume-opensearch-secrets
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: opensearch
    nodePublishSecretRef:
      name: azure

Then when the operator pod starts, these two secrets will get created. It would be possible to add secrets of all clusters to this secret provider class manifest and they will get created.

@swoehrl-mw
Copy link
Collaborator

@nilushancosta. To be honest this sounds like quite the hack and not like something the operator should support.
I can see two options: Either you find a way to get the secrets created some other way (does it have to be mounted to the operator pod or would it suffice to have some random pod mount it so the secret is created?), or the operator gets proper support for handling other secret stores (which would likely be a bigger implementation).

cc @prudhvigodithi @salyh

@nilushancosta
Copy link
Contributor Author

okay @swoehrl-mw . I will check if there is another way to do this.
It is not required to mount it to the operator pod itself. If we could create another pod at the same time, that should create the secret.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
untriaged Issues that have not yet been triaged
Projects
None yet
Milestone
No milestone
2 participants
@nilushancosta @swoehrl-mw