Resource limitation for the sidecar container on Autopilot

[bhack]

Looking at the default pytorch example in this repository I see some performance incompatibilities with the minimum autopilot resources request[1].
I think that we will have many problem allocating sidecar resources if we have these high min limits in autopilot.

gcs-fuse-csi-driver/examples/pytorch/train-job-pytorch.yaml

Lines 35 to 39 in 8a8d871

	annotations:
	gke-gcsfuse/volumes: "true"
	gke-gcsfuse/cpu-limit: "10"
	gke-gcsfuse/memory-limit: 40Gi
	gke-gcsfuse/ephemeral-storage-limit: 20Gi

[1]https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-resource-requests

[bhack]

Please consider also that Autopilot is officially the default and recommended GKE since April.

[bhack]

@songjiaxun Do you prefer to have this on https://issuetracker.google.com ?

[songjiaxun]

Thanks for the question. I admit that the pytorch example may not work in Autopilot clusters. I am actively working on the AI/ML application tests and will update the example yaml soon.

@bhack are you a Googler by any chance? Could you DM me with more context?

[bhack]

I've DM to you.

It is not only pytorch, It will not work any real DL scenario as the CPU limit on large nodes for the sidecard it will be MAX:
2 CPU and 14GB Memory.

[bhack]

I think we have regressed a bit here.

Now autopilot is going to accept unlimited/burstable resource on the sidecard: #61

But it "secretly" overriding with minimal resource.
This by an usability point of view it is very confusing as users have direct notification about this overriding so they could expect to work in a burstable context.

Manually scaling sidecar cpu resources it is going to not let the pod scheduling on Autopilot (E.g. >6000m on H100):

Violations details: {"[denied by autogke-no-node-updates]":["Operation on nodes with changes in addition to cordon is not allowed in Autopilot."]}
Requested by user: 'system:serviceaccount:gpu-operator:node-feature-discovery', groups: 'system:serviceaccounts,system:serviceaccounts:gpu-operator,system:authenticated'.```