Enhancing Credential Configuration in CRDs: Leveraging Environment Variables

[geoffrey1330]

Description:
This discussion centers around improving the management of credential configurations in Kubernetes Custom Resource Definitions (CRDs), particularly addressing the need to securely handle sensitive information like API keys within the CRD's YAML files. Currently, there's a requirement to place such credentials in the config_data section of an API definition within CRDs.

Objective:
The primary goal is to explore better ways to handle credentials and secrets in CRDs to enhance security and ease of use. It would be beneficial to discuss potential solutions and existing methods.

Proposed Approach:
One potential approach to consider is adopting a mechanism similar to how Kubernetes handles secrets and ConfigMaps in Deployments. This could include leveraging environment variables in CRD YAML files for secure credential management.

i.e

    env:
    - name: <env_name>
      valueFrom:
        secretKeyRef:
          name: <secret_name>
          key: <secret_data_key>

[jonathanfoster]

Hey @geoffrey1330, thanks for taking the lead on this. I definitely have use cases for this feature and I’ve been watching the issue for a while.

I like the idea of using the existing ConfigMap pattern in Deployments. This is definitely a more k8s native approach.

I would also consider maintaining compatibility with existing Tyk KV secrets storage so external KV stores can still be used. This would prevent any incompatibility issues when migrating from JSON API definitions.

Let me see if I can create some test API definitions based on my use cases.

[jonathanfoster]

Here's a use case I encounter quite often.

apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: httpbin-bearer
spec:
  name: httpbin-bearer
  use_keyless: true
  protocol: http
  active: true
  proxy:
    target_url: http://httpbin.org/bearer
    listen_path: /httpbin/bearer
    strip_listen_path: true
  version_data:
    default_version: Default
    not_versioned: true
    versions:
      Default:
        name: Default
        global_headers:
          authorization: $secret_env.httpbin_bearer_auth

[geoffrey1330]

Alright still need to have discussion with contributors on the best way to approach this

[geoffrey1330]

Hi @buraksekili I have created a PR for the api structure here

[buraksekili]

Thank you for raising this concern.

Presently, we employ Kubernetes Secret references for specific fields like PinnedPublicKeysRefs, which are not part of the typical API Definition within Tyk. When the operator detects these fields with populated values, it retrieves the corresponding secret from Kubernetes. Subsequently, it creates a secret within Tyk using Tyk APIs and uses the ID of the newly created secret in the APIDefinition.

To illustrate this operation more explicitly, here's a rough scenario:

# apidefinition.yaml
spec:
   fooRef: mysecret 

Upon applying this manifest, the operator:

• Retrieves the Kubernetes Secret named mysecret
• Uploads the content of mysecret to Tyk
• Obtains the ID of the secret created on Tyk (let's assume the ID is 123)
• Assigns .spec.foo to 123

Consequently, the resulting structure becomes:

# apidefinition.yaml
spec:
   fooRef: mysecret
   foo: 123

This flow allows your API to successfully utilize your Kubernetes secret.

The challenge I've observed while employing secrets in Config Data is due to its nature as an arbitrary JSON, as outlined here. As it lacks a JSON schema, we face difficulty in marshaling the structure into a Go structure to inherently comprehend Kubernetes references within it. While we could consider hacky solutions to support this concept, it might restrict the potential use cases of Config Data, which we aim to avoid. Thus, we are unable to directly reference a secret using fields like valueFrom in the Config Data. I'm open to discussing potential implementation ideas to address this constraint.

[geoffrey1330]

We could consider creating another field in the struct called Env to deal with handling secret and configMap data.
so we could have something

# apidefinition.yaml
spec:
   env:
   - name: <env_name>
      valueFrom:
          secretKeyRef:
              name: <secret_name>
              key: <secret_data_key>
   
   - name: <env_name>
      valueFrom:
         configMapKeyRef:
            name: <configmap_name>
            key: <configmap_data_key>

[buraksekili]

While I understand the logic behind suggesting the use of Env to update environment variables, the challenge lies in managing updates to Config Data.
Config Data, being an arbitrary JSON object, lacks a defined structure known to the Operator. Therefore, once users declare environment variables via Env, the task remains: how do we know the fields requiring updates in the raw JSON object?

Assume that I have following ApiDefinition CRs:

spec:
   config_data:
           firstField: something
           secondField:
                 env:
                   - name: <env_name>
                   ...          

spec:
   config_data:
           completelyDifferentField:
                 otherSubField:
                     env:
                     - name: <env_name>
                       ...          

So, users are free to declare config_data in any format. Now, my concern is updating such unstructured data.

[geoffrey1330]

We could introduce another struct in the config_data which will be optional

type MapStringInterfaceType struct {
    // +kubebuilder:pruning:PreserveUnknownFields
    unstructured.Unstructured `json:",inline"`
    
    // +optional
   Env *EnvVariable `json:"env,omitempty"`
}

[geoffrey1330]

Hi @buraksekili base on the look of things the env approach might not be the best solution.
But proposing a new approach which allow users to securely reference secrets in Kubernetes ApiDefinition configurations using $secret.<SecretName>.<SecretKey>, streamlining secret management by dynamically fetching values during runtime.

i.e
it fetches the value of this secret base on the secretName and secretKey reference in the yaml.

  ...
        global_headers:
          authorization: $secret.tykSecret.httpbin_bearer_auth

[geoffrey1330]

Yes I will Implement it using the approach you suggested once i get the go ahead from your end

[komalsukhani]

Great! Good to go then.

[geoffrey1330]

Just to be clear on the expectation. we reference the secret in the config_data and then retrieve this secret key and value, append it to the config_data as fields right?

[komalsukhani]

I would suggest to add a new field, something like config_data_ref, to accept secret name. Operator will read the secret and populate the config_data while sending a request to Tyk Gateway/Dashboard.

[geoffrey1330]

Hi @komalsukhani I have been out for while.
Will resume working on this proposed solution this week.

[geoffrey1330]

Hi @buraksekili i'm facing some difficulties try to deploy the operator for development purposes so i could test my implementation.
this doc doesn't work me

[buraksekili]

can you please raise a new issue with the details of problems you faced?

[geoffrey1330]

Hi @buraksekili what do you think about this new approach.

The Approach will address concerns related to parsing Unstructured data by using a recursive traversal approach to handle nested structures with values stored as interface{}. Additionally, it identifies secretRef by checking for the presence of a field with that name in the Unstructured data. Once secretRef is detected, it retrieves the secret based on the specified name and key within the secretRef.

[geoffrey1330]

Alright I will look into this to see how to inculcate it with the tyk-operator

[komalsukhani]

@geoffrey1330 How about we add a support that config data can be provided via a secret. So that we don't have to handle all this complex scenarios.

spec:
   config_data_ref: "config_data_secret"

And actual config data will be stored in the secret.

[geoffrey1330]

This sounds like a good approach. we then pull the secret key and value and inject into the config_data field.

[geoffrey1330]

Hi @komalsukhani could we consider your approach as the solution so i could start looking into how to implement this.

[komalsukhani]

@geoffrey1330 Let me confirm with the team today and I will post an update here.

[jonathanfoster]

What if this feature were implemented as a k8s KV store in Tyk Gateway that could read values from Secret resources? Secret values could be referenced in ApiDefinitions using $secret_k8s.namespace.secret_name.secret_key.

The benefits to this approach are:

• Secrets can be managed natively in k8s
• No API changes to ApiDefinitions kind
• Backwards compatibility with JSON API definitions

The main con to this approach is it requires multiple changes to Tyk Gateway (e.g., new k8s KV store, new labels in mw_url_rewrite.go, new k8s store config).

I'm curious if there's a better way to implement a new KV store that doesn't require so many code changes.

[buraksekili]

@jonathanfoster that's sound cool but as you said this needs change in Tyk Gateway, maybe it might be better to ask opinions about this on https://github.com/TykTechnologies/tyk

[komalsukhani]

@jonathanfoster Operator can talk to Tyk setup which is installed both inside and outside k8s environment.
If we implement feature in Gateway(Note: We will also need to add support in Dashboard), feature won't work if Gateway is installed outside k8s environment.

On the other hand if we add support in Operator, it can work with all the setups.