Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trident-operator.imageRegistry with custom registry not handled properly #772

Closed
domruf opened this issue Oct 11, 2022 · 10 comments · May be fixed by #774
Closed

trident-operator.imageRegistry with custom registry not handled properly #772

domruf opened this issue Oct 11, 2022 · 10 comments · May be fixed by #774
Labels
bug tracked

Comments

@domruf
Copy link

domruf commented Oct 11, 2022 •
edited

Describe the bug
Our on-prem clusters can't access registries on the internet like registry.k8s.io directly. We have mirrors in our local registry though.
So I though I use a custom values.yaml and set trident-operator.imageRegistry.
The problem is, the helm chart (and operator) does not handle the path of the images properly then.
If I set imageRegistry to 'myregistry.com/netapp' the netapp/* images are found but the registry.k8s.io/sig-storage/* images are not.
If I set imageRegistry to 'myregistry.com/sig-storage' the registry.k8s.io/sig-storage/* images are found but the netapp/* images are not.

IMHO since trident gets its images from 2 different registries, there should also be 2 imageRegistry options, one for registry.k8s.io and one for docker.io (dockerhub).

BTW I already have a quick hack which solves this problem. I added an additional trident-operator.netappImageRegistry to my values and patched the helm chart like this:

diff --git a/helm/trident-operator/templates/_helpers.tpl b/helm/trident-operator/templates/_helpers.tpl
index a2974dcd..483a692a 100644
--- a/helm/trident-operator/templates/_helpers.tpl
+++ b/helm/trident-operator/templates/_helpers.tpl
@@ -68,8 +68,8 @@ Trident operator image
 {{- define "trident-operator.image" -}}
 {{- if .Values.operatorImage }}
 {{- .Values.operatorImage }}
-{{- else if .Values.imageRegistry }}
-{{- .Values.imageRegistry }}/trident-operator:{{ .Values.operatorImageTag | default .Chart.AppVersion }}
+{{- else if .Values.netappImageRegistry }}
+{{- .Values.netappImageRegistry }}/trident-operator:{{ .Values.operatorImageTag | default .Chart.AppVersion }}
 {{- else }}
 {{- "" }}docker.io/netapp/trident-operator:{{ .Values.operatorImageTag | default .Chart.AppVersion }}
 {{- end }}
@@ -113,8 +113,8 @@ Trident AutoSupport image
 {{- define "trident.autosupportImage" -}}
 {{- if .Values.tridentAutosupportImage }}
 {{- .Values.tridentAutosupportImage }}
-{{- else if .Values.imageRegistry }}
-{{- .Values.imageRegistry }}/trident-autosupport:{{ .Values.tridentAutosupportImageTag | default .Chart.AppVersion | trunc 5}}
+{{- else if .Values.netappImageRegistry }}
+{{- .Values.netappImageRegistry }}/trident-autosupport:{{ .Values.tridentAutosupportImageTag | default .Chart.AppVersion | trunc 5}}
 {{- else }}
 {{- "" }}docker.io/netapp/trident-autosupport:{{ .Values.tridentAutosupportImageTag | default .Chart.AppVersion | trunc 5}}
 {{- end }}
@@ -148,8 +148,8 @@ Trident image
 {{- define "trident.image" -}}
 {{- if .Values.tridentImage }}
 {{- .Values.tridentImage }}
-{{- else if .Values.imageRegistry }}
-{{- .Values.imageRegistry }}/trident:{{ .Values.tridentImageTag | default .Chart.AppVersion }}
+{{- else if .Values.netappImageRegistry }}
+{{- .Values.netappImageRegistry }}/trident:{{ .Values.tridentImageTag | default .Chart.AppVersion }}
 {{- else }}
 {{- "" }}docker.io/netapp/trident:{{ .Values.tridentImageTag | default .Chart.AppVersion }}
 {{- end }}

With these changes trident runs fine so far.

Environment
Provide accurate information about the environment to help us reproduce the issue.

  • Trident version: [22.07]
  • Trident installation flags used: []
  • Container runtime: [docker 20.10.10]
  • Kubernetes version: [1.21.7]
  • Kubernetes orchestrator: [Rancher v2.6.3]
  • Kubernetes enabled feature gates: []
  • OS: [Ubuntu 18.04]
  • NetApp backend types: [ONTAP]
  • Other:

To Reproduce
Steps to reproduce the behavior:
Deploy trident via helm and set trident-operator.imageRegistry to a custom registry.

Expected behavior
After installing the trident helm chart, all containers are running properly.

Additional context
Add any other context about the problem here.
@domruf domruf added the bug label Oct 11, 2022
@gnarl gnarl added the tracked label Oct 11, 2022
@mayrstefan mayrstefan mentioned this issue Oct 18, 2022
Open
@mayrstefan
Copy link

The fact that for an offline deployment all images have to be available in one registry can be found in https://docs.netapp.com/us-en/trident/trident-get-started/kubernetes-deploy.html#offline-deployment.
But because it is quite common to have registry mirrors (or pass through proxies) I'd prefer to be able to configure them individually as requested by the issue author.

@domruf
Copy link
Author

domruf commented Oct 25, 2022

My problem is that I have put the images in one registry. In the description above I called the registry 'myregistry.com'.
But the path part of the image is not handled properly at the moment.
So I can configure it to successfully get the sig-storage or the netapp images. But I see no way to configure it to get both.

@rohit-arora-dev
Copy link
Contributor

You may also try to use the tridentImage and tridentAutosupportImage options (complete path to your Trident and autosupport images) along with the imageRegistry option pointing to the sig-storage image registry.

@domruf
Copy link
Author

domruf commented Oct 26, 2022

Using all 3 values

  imageRegistry: myregistry.com/sig-storage
  tridentImage: myregistry.com/netapp/trident:22.07.0
  autosupportImage: myregistry.com/netapp/trident-autosupport:22.07.0

still results in an error, because it then tries to use myregistry.com/sig-storage/trident-operator:22.07.0
But the correct image would be myregistry.com/netapp/trident-operator:22.07.0

@rohit-arora-dev
Copy link
Contributor

Right, I missed including the Operator Image in my response. So if you supply these values:

  imageRegistry: myregistry.com/sig-storage
  tridentImage: myregistry.com/netapp/trident:22.07.0
  operatorImage: myregistry.com/netapp/trident-operator:22.07.0
  autosupportImage: myregistry.com/netapp/trident-autosupport:22.07.0

Are you able to install Trident successfully?

@mayrstefan
Copy link

From my understanding it should be:

imageRegistry: myregistry.com/sig-storage
tridentImage: myregistry.com/netapp/trident:22.07.0
operatorImage: myregistry.com/netapp/trident-operator:22.07.0
autoSupportImage: myregistry.com/netapp/trident-autosupport:22.07

For autoSupport the version tag is truncated to 5 characters in the helm chart.

If you put all images into a local registry they all need to be put into the same path.
Using imageRegistry: myregistry.com/mypath should result in the following image names:

  • myregistry.com/mypath/trident:22.07.0
  • myregistry.com/mypath/trident-operator:22.07
  • myregistry.com/mypath/trident-autosupport:22.07
  • myregistry.com/mypath/csi-provisioner:v3.3.0
  • myregistry.com/mypath/csi-attacher:v4.0.0
  • myregistry.com/mypath/csi-resizer:v1.6.0
  • myregistry.com/mypath/csi-snapshotter:v6.1.0 or v3.0.3
  • myregistry.com/mypath/csi-node-driver-registrar:v2.5.1
  • myregistry.com/mypath/livenessprobe:v2.5.0

That doesn't work with most typical pass-through proxies. We then need to manually mirror images into an internal registry. That's why it is worth to change the helm chart and cli to accept two registry names.

@domruf
Copy link
Author

domruf commented Oct 27, 2022 •
edited

@ntap-arorar

  imageRegistry: myregistry.com/sig-storage
  tridentImage: myregistry.com/netapp/trident:22.07.0
  operatorImage: myregistry.com/netapp/trident-operator:22.07.0
  autosupportImage: myregistry.com/netapp/trident-autosupport:22.07.0

This did actually works. At least I see not problems ATM after upgrading my helm chart release.

But I think we can agree that this is not really intuitive. And I believe it is not documented this way either.
So the whole thing should be improved upon.

@domruf
Copy link
Author

domruf commented Oct 27, 2022

I must correct myself. The mentioned settings did not work after all.
Even though I set autosupportImage: myregistry.com/netapp/trident-autosupport:22.07.0, the pod of the 'trident-csi' deployment tries to use myregistry.com/sig-storage/trident-autosupport:22.07, which naturally fails.

I guess I did not notice this at first because the pod did not get updated directly after I upgraded the helm chart.

@domruf
Copy link
Author

domruf commented Oct 27, 2022 •
edited

I gave the helm chart sources another look and found out the value is called tridentAutosupportImage not autosupportImage.
Now I use

  imageRegistry: myregistry.com/sig-storage
  tridentImage: myregistry.com/netapp/trident:22.07.0
  operatorImage: myregistry.com/netapp/trident-operator:22.07.0
  tridentAutosupportImage: myregistry.com/netapp/trident-autosupport:22.07.0

and it seems to work.

@gnarl
Copy link
Contributor

gnarl commented Feb 1, 2023

Hi @domruf,

New offline install guides are included in the Trident documentation for offline operator and Helm installs.

Since not every Trident customer environment is the same, the offline install process has the flexibility of using either one registry location for all required images or multiple registry locations.

@gnarl gnarl closed this as completed Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug tracked
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add tridentImageRegistry to Helm chart (trident-image-registry to cli) mayrstefan/trident
4 participants
@gnarl @domruf @mayrstefan @rohit-arora-dev