Coordinator with K8s ingress

[youjeongsue]

Hello!

I'm deploying and using Uniffle 0.7.1 on kubernetes.

But what I want to do is to run pyspark(with uniffle) outside of kubernetes.

spark_conf = SparkConf() \
     ...
    .set("spark.rss.enabled", "true") \
    .set("spark.shuffle.manager", "org.apache.spark.shuffle.RssShuffleManager") \
    .set("spark.rss.coordinator.quorum", "rss-coordinator-service-0.our-domain.com:443,rss-coordinator-service-1.our-domain.com:443") \ <- here is the point
    .set("spark.rss.storage.type", "MEMORY_LOCALFILE")

The problem is that we can't access it in the form {svc}.{namespace}:19997 like we can in an in-cluster.

We tried ingress with a grpc setup, but it didn't work. (Below is the error log)

annotations:
  nginx.ingress.kubernetes.io/backend-protocol: GRPC

23/09/13 03:56:06 WARN ShuffleWriteClientImpl: Fail to get conf from Coordinator grpc client ref to rss-coordinator-service-0.our-domain.com:443
23/09/13 03:56:06 WARN ShuffleWriteClientImpl: Fail to get conf from Coordinator grpc client ref to rss-coordinator-service-1.our-domain.com:443

Maybe you can help me figure out how to achieve what I want?

[jerqi]

It' better to use host network. It will be better that the Spark cluster can connect the Uniffle cluster, otherwise we need to transport huge network data. Now, Uniffle operator don't support annotations, too.
Maybe you should paste the log of Ngnix, do the Spark application access the nginx of ingressing?

[youjeongsue]

First of all, thanks for the kind words! The recommended practice is to use the host ip as it is. I understand.

But this ip changes when the coordinator pod is rescheduled, so we want to get a fixed address.
This is a problem that needs to be solved on the Kubernetes side, am I right..?

[jerqi]

We use DNS instead of ip. K8S should have similar concept. We choose host network instead of virtual network to let the external host access the K8S cluster.