Security Attributes for Nginx Ingress

[3rk1n]

Hi,

I want to close security recommendations for Kubernetes Cluster in Microsoft Defender for Cloud.

One of them is "Kubernetes clusters should disable automounting API credentials" and it can be solved by added "automountServiceAccountToken: false" for POD security "spec --> template --> spec"

spec:
  template:
    metadata:
    spec:
      automountServiceAccountToken: false

But when I am adding this value for the Ingress Nginx Controller, the pods give me CrashLoopBackOff status and that logs in below.

How can I solve this?

client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
client_config.go:623] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
main.go:267] Error while initiating a connection to the Kubernetes API server. This could mean the cluster is misconfigured (e.g. it has invalid API server certificates or Service Accounts configuration). Reason: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

[vepatel]

Hi @3rk1n, from you problem statement I understand that this is a security recommendation from Microsoft Azure Defender but Ingress Controller needs credentials to access the K8s API in order to function.

So the behaviour you're seeing after setting automountServiceAccountToken: false is completely expected. Our recommendation would be either:

• Ignore this particular security recommendation and use rbac security measures to prevent unwanted access OR
• Set automountServiceAccountToken: false and then immediately manually configure the API access token.

See: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server

[brianehlert]

As vepatel stated, while the Azure Defender guidelines are good and intended to cover containers from a generic sense, it does not realize that any "controller" or "operator" are special Kubernetes tools applications that require access to the Kubernetes API.
And the way to grant API access to this applications is for Kubernetes itself to grant a token to the container.

As I stated, the guidance of Azure Defender is prudent from an entirely generic perspective. For example, your web server application in a container does not need access to the API. But your ingress controller running in a container does require access to the API - to enumerate configurations, backend service endpoints, write events back, and so forth.

Please let us know if you need further guidance as you define the necessary exception for this application.

[3rk1n]

Thank you for your replies, vepatel and brianehlert.

At this point, as vepatel referred a documentation, if I can define a generic serviceaccount and set automountserviceAccountToken to false for it, and of course I'll have to authorize it as well, the system works properly?

[vepatel]

Hi @3rk1n I'd still advise against the approach of having a manually managed SA token as IC needs to communicate with API at all the times. If the token expires, it may result in an outage.

[brianehlert]

You will know pretty quickly if it works or not. As you report, the controller process will not start without being able to access the K8s API.
Other Azure customers have taken our guidance and defined an exception due to the unique characteristics of this particular application.

Like I said, 99% of the time the rule applies as it is written for general services that run in Kubernetes that do not interact with the K8s API.
But as an infrastructure tool, no different from cert-manager, external-dns, Prometheus, and so forth, there is a need to interact with the API. And K8s controls the tokens, token rotation, and whatnot automatically on a service by service basis.

[3rk1n]

Thank you for your help, and detailed explanation.