Exposing InferenceService

[israel-hdez]

I posted t his question on Slack, but I didn't get any reply. So, I'm re-posting here, hoping an answer:

I see that when creating an InferenceService, some KNative services are created to support it. I see that KNative creates its own Routes and Istio configurations. Then, once KNative has setup routes, KServe creates its own configurations, based on KNative routes by removing some suffixes and finally sets a URL in the Status of the InferenceService. I see that KServe just creates some VirtualServices and binds them to Gateways.

Is it OK to think that KServe assumes that whatever *.domain being used, is configured to route requests to Istio?

I'm trying to understand how to correctly expose an InferenceService. The only hint I have right now is the DNS configuration of KNative, but I'm not sure if KServe adheres-to/reuses/assumes that DNS configuration to expose InferenceServices.

[elukey]

Hi! From my experience (may be incomplete) - the Istio Ingress is instructed to route requests checking the HTTP Host header, so that is the place where you should configure your domain data. Do you have any specific scenario in mind?

[israel-hdez]

The question is more about exposing the InferenceService:

Client from         Istio              KServe
     +----------------+-------------------+->
 Internet       IngressGateway    InferenceService

The IngressGateway -> InferenceService part is clear to me. The internet -> IngressGw part is where I wanted to learn more.

In KServe website, there is no mention about how to expose (or not expose) an InferenceService outside the cluster. After some trials, my perception is that KServe fully relies on KNative and its DNS configuration to expose services. I wanted to know if this is right, or if I'm misunderstanding?

---

More background:

My setup for the first part is more like the following:

Client from         Cluster            Istio
     +----------------+-------------------+---------> Etc
 Internet       IngressService      IngressGateway

All traffic that wants to go into the cluster needs to go through the Cluster IngressService. That Cluster Ingress does not accept wildcard domains (i.e. *.example.com is not valid configuration). For KNative+Istio, we have had to configure each individual domain we expect so that traffic is forwarded to Istio's IngressGateway.

Despite this may sound inconvenient, IMHO, it adds some security, because traffic to unknown hostnames is discarded at the border. However, after trying to incorporate KServe it means that out-of-the-box the URL at the InferenceService does not work. I've successfully configured the Cluster Ingress to accept traffic of the hostname stated in the InferenceService and it works. Understanding KServe expectations will help me to know if this is the right way to configure the Cluster Ingress.

[elukey]

Not sure if I got your use case 100%, but it sounds like you have a sort-of api-gateway between clients and the Istio IngressGateway. This is similar to the config that we run:

Client from       Cluster             LB VIP           Istio
     +----------------+-------------------+----------------+---------> Etc
 Internet       API-Gateway     (Linux LVS)    IngressGateway

So your equivalent of IngressService gets a request for (in our case, api.blabla.org/etc..) and proxies the request to LB-VIP (keeping the same URI, with a little bit of changes to it) setting the right HTTP Host header (namely, the one that you mentioned before). The Istio Gateway accepts the request and routes it to the correct pod.

The LB VIP is just Linux LVS configured to route requests (L4 LB) to the Istio Gateway pods (they expose their port via nodePort config).

Hope that helps!

[israel-hdez]

It helps.

In our case, if I'm not wrong, that Cluster IngressService is based on HAProxy. If forwarding traffic like you do is something OK to do, then I think we are also on a good path for exposing the InferenceServices.

Thanks for sharing! It makes my concerns to go away.