Can operator-sdk run bundle use local image?

[whg517]

Type of question

Hi, I am writing operator using operator-sdk. Recently I was looking for a solution on how to do e2e testing. I chose to use kind as a temporary test environment.

I started the kind node locally, and then built the operator image and bundle image locally. I used kind load docker-image to load the two images to the kind cluster. When I tried to initialize the test suite environment through operator-sdk run bundle before the test, I found that operator-sdk would pull the images through the network.

In CI environment, these operations are temporary, in order to verify that there is no problem with my logic, I think operator-sdk should first try to use the local bundle image, instead of pulling the remote repository, otherwise you need to provide a test repository to store the temporary image or special version. I think this mechanism is bad for development testing.

I want to ask is this something wrong with me or does operator-sdk run bundle always pull the image of the remote repository?

Question

What did you do?

• make bundle
• make bundle-build
• kind load docker-image quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0
• operator-sdk run bundle quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0

output:

INFO[0001] trying next host - response was http.StatusNotFound  host=quay.io
FATA[0001] Failed to run bundle: pull bundle image: error pulling image quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0: error resolving name for image ref quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0: quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0: not found 

What did you expect to see?

operator-sdk shoule use local image builded with docker, rather then pull bundle.

What did you see instead? Under which circumstances?

above

Environment

Operator type:

golang

Kubernetes cluster type:

$ operator-sdk version

operator-sdk version: "v1.33.0", commit: "542966812906456a8d67cf7284fc6410b104e118", kubernetes version: "v1.27.0", go version: "go1.21.5", GOOS: "darwin", GOARCH: "arm64"

$ go version (if language is Go)

go version go1.21.5 darwin/arm64

$ kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:13:53Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:38:14Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/arm64"}

Additional context

[whg517]

By reading the source code, I found that the mirror implementation logic supports getting the image from the local, but the local mirror logic is not checked at the time of calling.

operator-sdk/internal/registry/image.go

Lines 30 to 72 in 4abd483

	func ExtractBundleImage(ctx context.Context, logger *log.Entry, image string, local bool, skipTLSVerify bool, useHTTP bool) (string, error) {
	if logger == nil {
	logger = DiscardLogger()
	}
	// Use a temp directory for bundle files. This will likely be removed by
	// the caller.
	wd, err := os.Getwd()
	if err != nil {
	return "", err
	}
	bundleDir, err := os.MkdirTemp(wd, "bundle-")
	if err != nil {
	return "", err
	}
	// This should always work, but if it doesn't bundleDir is still valid.
	if dir, err := filepath.Rel(wd, bundleDir); err == nil {
	bundleDir = dir
	}
	// Export the image into bundleDir.
	logger = logger.WithFields(log.Fields{"dir": bundleDir})
	// Use a containerd registry instead of shelling out to a container tool.
	reg, err := containerdregistry.NewRegistry(
	containerdregistry.WithLog(logger),
	containerdregistry.SkipTLSVerify(skipTLSVerify),
	containerdregistry.WithPlainHTTP(useHTTP))
	if err != nil {
	return "", err
	}
	defer func() {
	if err := reg.Destroy(); err != nil {
	logger.WithError(err).Warn("Error destroying local cache")
	}
	}()
	// Pull the image if it isn't present locally.
	if !local {
	if err := reg.Pull(ctx, registryimage.SimpleReference(image)); err != nil {
	return "", fmt.Errorf("error pulling image %s: %v", image, err)
	}
	}

But at the time of the call, no local judgment is made

operator-sdk/internal/olm/operator/helpers.go

Lines 37 to 40 in 4abd483

	func LoadBundle(ctx context.Context, bundleImage string, skipTLSVerify bool, useHTTP bool) (registryutil.LabelsMap, *apimanifests.Bundle, error) {
	bundlePath, err := registryutil.ExtractBundleImage(ctx, nil, bundleImage, false, skipTLSVerify, useHTTP)
	if err != nil {
	return nil, nil, fmt.Errorf("pull bundle image: %v", err)

I was wondering if anyone could fix this?

[burmanm]

I recommend installing local registry like instructed here: https://kind.sigs.k8s.io/docs/user/local-registry/

After that, you can push to that registry and run a local bundle. For example, my quick test:

➜  cass-operator git:(master) ✗ bin/operator-sdk run bundle localhost:5001/k8ssandra/cass-operator-bundle:v1.20.0-dev.faa4d28-20240404 --use-http
INFO[0008] Creating a File-Based Catalog of the bundle "localhost:5001/k8ssandra/cass-operator-bundle:v1.20.0-dev.faa4d28-20240404" 
INFO[0008] Generated a valid File-Based Catalog         
INFO[0016] Created registry pod: 001-k8ssandra-cass-operator-bundle-v1-20-0-dev-faa4d28-20240404 
INFO[0016] Created CatalogSource: cass-operator-catalog 
INFO[0016] OperatorGroup "operator-sdk-og" created      
INFO[0016] Created Subscription: cass-operator-v1-20-0-dev-faa4d28-20240404-sub 
INFO[0023] Approved InstallPlan install-kbnqd for the Subscription: cass-operator-v1-20-0-dev-faa4d28-20240404-sub 
INFO[0023] Waiting for ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" to reach 'Succeeded' phase 
INFO[0024]   Waiting for ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" to appear 
INFO[0026]   Found ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" phase: Pending 
INFO[0028]   Found ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" phase: Installing 
INFO[0089]   Found ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" phase: Succeeded 
INFO[0089] OLM has successfully installed "cass-operator.v1.20.0-dev.faa4d28-20240404" 
➜  cass-operator git:(master) ✗

Personally I always have local registry installed in my kind setups, since it gives more "e2e" feeling in that sense.