[BUG] Failed to pull image "gcr.io/spark-operator/spark:v3.1.1"

[sidi-elwely]

Description

Unable to create a spark-application

• ✋ I have searched the open/closed issues and my issue is not listed.

Steps to reproduce the behavior:

1. Set up a new kubenetes cluster. I set up one in gcloud.
2. Get kubenetes cluster config
3. Install spark-operator by helm
4. create spark-pi.yaml by using (https://github.com/kubeflow/spark-operator/blob/master/examples/spark-pi.yaml)
5. Apply the file

Actual behavior

Pod in status Failed because of ImagePullBackOff

Terminal Output Screenshot(s)

Events:
Type Reason Age From Message

---

Normal Scheduled 12s default-scheduler Successfully assigned default/spark-pi-driver to kind-control-plane
Warning FailedMount 10s (x3 over 11s) kubelet MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-cfc48c8f3d591c55-conf-map" not found
Normal Pulling 4s kubelet Pulling image "gcr.io/spark-operator/spark:v3.1.1"
Warning Failed 1s kubelet Failed to pull image "gcr.io/spark-operator/spark:v3.1.1": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/spark-operator/spark:v3.1.1": failed to resolve reference "gcr.io/spark-operator/spark:v3.1.1": gcr.io/spark-operator/spark:v3.1.1: not found
Warning Failed 1s kubelet Error: ErrImagePull
Normal BackOff 1s kubelet Back-off pulling image "gcr.io/spark-operator/spark:v3.1.1"
Warning Failed 1s kubelet Error: ImagePullBackOff

Environment & Versions

• Spark Operator App version: 1.4.5
• Helm Chart Version: v3.14.4
• Kubernetes Version: v1.27.3
• Apache Spark version: 3.5.0

[networkingana]

I have the same issue, what image should we use?

[peter-mcclonski]

Good morning,

As of #2010, the examples have been updated to reference the official spark image available on dockerhub: spark:3.5.0. Unfortunately, the legacy images are no longer available. Fortunately, the official spark images are fully compatible with this operator.

[sidi-elwely]

Thank you for the information, but also in spar-pi-prometheus we have the same problem, this image: gcr.io/spark-operator/spark:v3.1.0-gcs-prometheus does not work. could you please help us!

[peter-mcclonski]

@sidi-elwely #2010 didn't update the prometheus-enabled image, which currently is not published by any of the CI jobs. I'll defer to a maintainer as to whether this is something worth re-enabling, but I think it's likely to need some rework regardless. Right now the image is tied specifically to GCP, which I'm comfortable saying isn't optimal. The meat of the image WRT prometheus is a single jar and a couple of conf files-- perhaps not worth maintaining as a separate image, but I can imagine a few ways to ease usage.

[peter-mcclonski]

See https://github.com/kubeflow/spark-operator/tree/master/spark-docker if you're interested in creating your own prometheus-enabled image in the mean time.