Load balancing across namespaces

[romanchyla]

It would be terrific if one could use haproxy to load balance services that live in different namespaces. I'll describe our particular use-case, there are however others (many many upvotes!), see this: kubernetes/kubernetes#17088

In our particular case we have 3 deployment stages; inside one K8 cluster, logically separated through namespaces (DEV, QA, PROD). DEV is used for testing of new releases; QA is for quality control - only vetted releases make it there. PROD is the same as QA but gets the real traffic.

So: each of these 3 namespace contains identical set of services; these mostly differ in the amount of traffic they handle (limits on pods they are allowed to grow). This model is working nicely and is probably not uncommon.

One of our services is particularly challenging to scale up/down quickly (to use proverbial cats vs cattle; we'd like to treat it as cattle). It is the search engine with large index (if populated from storage - takes 30mins to copy). When the search engine starts, it warms up (loads indexes, executes queries, builds additional data structures etc) - so it literally can take 15mins before a new pod is built from scratch (we do use persistent volumes etc)

That's why we always hold multiple instances of the search engine ready, even if traffic does not warrant it. And the opposite happens too. They got overwhelmed with requests. Our solution so far was to redirect traffic to QA namespace (change configuration of k8 Service; point it to different ingress DNS)

This is however not automated. So bad stuff happens until someone get's to it...

We can't bring new pods fast enough, but we already have multiple instances of the search service living in other namespaces. So it would make lot of sense to fall back on the QA namespace from PROD (when disaster strikes; or even when the load is too high -- that's a judgement call)

K8 ingress controller cannot do it -- they decided (so far) to keep things logically separated. It makes sense. They follow certain philosophy. However, it would also make lot of sense to allow ingress to reach across namespaces -- I believe there is real need for a solution and haproxy kubernetes ingress seems like a great alternative that could fill the gap.

[NickMRamirez]

You can use ingress.class to target a "production" controller. For example, if you have an ingress controller with a class of "production", then the QA ingress could change its ingress.class annotation to point to this, shifting that ingress to production. The section An Ingress Controller that You Target in this blog post shows the concept.

UPDATE: If I create two different Ingress objects that target different Services, but both target the same ingress.class, then they will both be added to the HAProxy configuration, but in different backends that match the name of the service. So, they won't be load balanced together, making my proposed solution not work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.