Cross-namespace Ingress

[paralin]

As far as I can tell right now it's only possible to create an ingress to address services inside the namespace in which the Ingress resides. It would be good to be able to address services in any namespace.

It's possible that I'm missing something and this is already possible - if so it'd be great if this was documented.

[bprashanth]

Nope, not allowing this was a conscious decision (but one that I can be convinced against). Can you describe your use case? The beta model partitions users on namespace boundaries and disallows service sharing across namespaces. You might argue that you want a single loadbalancer for the entire cluster, to which I ask, what is in the namespaces? i.e why not use 1 namespace if you want to allow sharing.

[paralin]

@bprashanth I'm running multiple projects on a cluster - kubernetes tests, blog, API for a project. I want to address these as subdomains on my domain using a single ingress controller because load balancers and IP addresses are expensive on GCE.

[liggitt]

It would be good to be able to address services in any namespace.

It was intentionally avoided. Cross namespace references would be a prime source of privilege escalation attacks.

cc @kubernetes/kube-iam

[paralin]

I'll close this for now, makes sense.

[thockin]

FWIW you can set up a Service in namespace X with no selector and a
manual Endpoints that just lists another Service's IP. It's yet another
bounce, but it seems to work. :)

On Thu, Nov 12, 2015 at 5:14 PM, Jordan Liggitt notifications@github.com
wrote:

It would be good to be able to address services in any namespace.

It was intentionally avoided. Cross namespace references would be a prime
source of privilege escalation attacks.

cc @kubernetes/kube-iam
https://github.com/orgs/kubernetes/teams/kube-iam

—
Reply to this email directly or view it on GitHub
#17088 (comment)
.

[krancour]

I would tend to imagine the use case that @paralin described is common. I'm looking at an ingress controller as a system component and a means of reflecting any service in the cluster to the outside world. Running one (perhaps even in the kube-system namespace) that can handle ingress for all services just seems to make a lot of sense.

[bprashanth]

Cross namespace references would be a prime source of privilege escalation attacks.

That depends on what you have in your namespaces right (which is why i asked for clarification)? Isn't it only risky IFF you're partitioning users across a namespace security boundry?

[bprashanth]

There seems to be demand for cross namespace ingress x service resolution. We should at least reconsider.

[erictune]

I think we want some kind of admission controller which does:

if ! req.Kind = "Ingress" { return }
ingress  := req.AsIngress()
for each serviceRef field in ingress {
  if req.User is not Authorized() to modify the service pointed to by serviceRef {
    reject request
  }
}

Then, to modify an ingress you have to have and owner-like permission on all the services it targets.

[paralin]

Might be good to revisit this now in 2016 :)

[bprashanth]

@kubernetes/kube-iam thoughts/volunteers to implement an admission controller? Do we authorize based on the user field of a request today or is that unprecedented?

[liggitt]

to modify an ingress you have to have and owner-like permission on all the services it targets.

I think I'd want some record or indication of the cross-namespace relationship to exist, so the targeted service could know it was exposed. I want to avoid the scenario where someone had access to a service (legitimately or otherwise), set up ingress from other namespaces, then had their access removed and continued accessing the services without the service owner's awareness.

Do we authorize based on the user field of a request today or is that unprecedented?

The authorization layer is based on the user info on a request. This would be the first objectref authorization I know of.

[erictune]

@liggitt's raises some good concerns. I broke them down into two cases when thinking about them.

1. assuming everyone is trustworthy, it might still be hard to reason about the network security of a service just by looking at the service object (or just by looking at objects in the same namespace). IT might be misconfigured.
   • I agree with this, to a point
   • however, creating an object that represents a connection between two services seems like it would scale poorly.
   • we need a solution that scales with the number of services, not the number of interconnections, I think.
2. assuming there is someone not-trustworthy, they can misconfigure they network in a way where the misconfiguration persists after some of their access is revoked.
   • Yes. But we have this problem worse with pods, configmap, etc. The bad actor might have run pods that that are doing the wrong thing, and auditing this is very hard.

[thockin]

Is this moving into the topic of micro-segmentation and access policy?

On Wed, Jan 20, 2016 at 2:19 PM, Eric Tune notifications@github.com wrote:

@liggitt https://github.com/liggitt's raises some good concerns. I
broke them down into two cases when thinking about them.

1. assuming everyone is trustworthy, it might still be hard to reason
   about the network security of a service just by looking at the service
   object (or just by looking at objects in the same namespace). IT might be
   misconfigured.
   • I agree with this, to a point
   • however, creating an object that represents a connection between
     two services seems like it would scale poorly.
   • we need a solution that scales with the number of services, not
     the number of interconnections, I think.
2. assuming there is someone not-trustworthy, they can misconfigure
   they network in a way where the misconfiguration persists after some of
   their access is revoked.
   • Yes. But we have this problem worse with pods, configmap, etc.
     The bad actor might have run pods that that are doing the wrong thing, and
     auditing this is very hard.

—
Reply to this email directly or view it on GitHub
#17088 (comment)
.

[erictune]

yes.

[ghost]

The two main models proposed for network segmentation are:

1. decorate Services (and maybe Pods) with a field indicting "allow-from".
   This basically allows one to draw the directed graph of an application,
   sort of.

2. implement a "policy group" object which selects Pods to which to apply
   policy, and includes some simple policy statements like "allow-from"

On Thu, Jan 21, 2016 at 11:52 PM, Tim Hockin notifications@github.com
wrote:

Is this moving into the topic of micro-segmentation and access policy?

On Wed, Jan 20, 2016 at 2:19 PM, Eric Tune notifications@github.com
wrote:

@liggitt https://github.com/liggitt's raises some good concerns. I
broke them down into two cases when thinking about them.

1. assuming everyone is trustworthy, it might still be hard to reason
   about the network security of a service just by looking at the service
   object (or just by looking at objects in the same namespace). IT might be
   misconfigured.
2. I agree with this, to a point
3. however, creating an object that represents a connection between
   two services seems like it would scale poorly.
4. we need a solution that scales with the number of services, not
   the number of interconnections, I think.
5. assuming there is someone not-trustworthy, they can misconfigure
   they network in a way where the misconfiguration persists after some of
   their access is revoked.
6. Yes. But we have this problem worse with pods, configmap, etc.
   The bad actor might have run pods that that are doing the wrong thing,
   and
   auditing this is very hard.

—
Reply to this email directly or view it on GitHub
<
#17088 (comment)

.

—
Reply to this email directly or view it on GitHub
#17088 (comment)
.

[erictune]

@thockin what issue does one go to to learn more and comment?

[thockin]

It's being discussed in the network SIG mailing list as we haggle over
multitudes of ideas and whittle it down to a few viable ones.

Start here:

https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit

One proposal:

https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit

Another is in email:

https://groups.google.com/forum/#!topic/kubernetes-sig-network/Zcxl0lfGYLY

On Fri, Jan 22, 2016 at 12:46 PM, Eric Tune notifications@github.com
wrote:

@thockin https://github.com/thockin what issue does one go to to learn
more and comment?

—
Reply to this email directly or view it on GitHub
#17088 (comment)
.

[erictune]

Talked to @thockin and @bprashanth
It sounds like the Ingress resource may undergo some refactoring this quarter, possibly splitting into two objects. We should revisit ingress security when those discussion happen.

[wstrange]

This would be a nice feature to have. For example, if you want a pseudo multi-tenant solution - with each tenant running in a separate namespace. The ingress could do hostname based routing to the right backend namespace. ${tenant}.example.com -> service "foo" in namespace ${tenant}

I suppose one can do this today on GKE, but I gather you end up with one HTTP load balancer per namespace - which could get quite expensive and seems unnecessary.

[jimmycuadra]

This limitation throws a big wrench in how my company was planning to use ingresses. Our use case is running multiple copies of the same application stack at different versions, and to keep the stacks isolated from each other, we use namespaces. We'd planned to run a single ingress controller that knows how to determine which applications and versions of those applications by the subdomain of the incoming requests.

The reason for using namespaces for isolating these stacks are:

1. To be extra safe about not having applications interfere with each other based on what else happens to be running in the cluster or from similarly named services.
2. To get around name collisions for services. It's not possible to have two services in the same namespace with the same name, so application dependencies like "redis" or "mysql" need to be in different namespaces to use those simple names without faking a namespace by changing the name of the service.

See my unanswered Stack Overflow question, Kubernetes services for different application tracks, for more details on our use case.

Our ingress controller is exposed to the outside world via NodePort (80 and 443), and we have an ELB in AWS pointing at the whole cluster. With the namespace restriction for ingresses, we'd need one ingress controller per namespace and there would be no way to have a single ELB forwarding ports 80 and 443 to the cluster.

[RobertoDonPedro]

Is there currently a solution or workaround availble that works on GKE?
I tried the externalName approach with both gce-ingress as well as nginx-ingress, but in any case I end up with

error while evaluating the ingress spec: service "..." is type "ExternalName", expected "NodePort" or "LoadBalancer"

[cabrinoob]

Using Istio allows you this kind of things (among others). You can create a "selector-less" service in your Ingress namespace and point your ingress on it. Then, you create a VirtualService that can point to another namespace service.

[Erokos]

@spenceclark
Did you name the ingress in your other namespace the same as the main one or different? Are you maybe on AWS?

EDIT: I managed to describe the behaviour you described by creating an ingress per workspace and tieing to the same host, i.e. AWS ELB dns name. It somehow works.

[tymokvo]

Just wanted to +1 implementing this.
Our use case is multiple staging versions of an application (one of the use cases the docs suggest for namespaces) that all have a new database that we would like to isolate from one another.

It would be great if we could use one load balancer/ingress to point to services in different namespaces to isolate the versions, but use one TLS cert, one DNS record, etc on the load balancer. It could even require an annotation like app.kubernetes.io/ingress.allow-across-namespace: true with a big red warning in the docs that it is a potential security risk.

[onpaws]

@Erokos

Did you name the ingress in your other namespace the same as the main one or different? Are you maybe on AWS?

On Azure, k8s 1.18, single node in case it matters (this is my 'test ideas out' cluster).
I created three different Ingress resources with different names, one in ingress-basic, and two more (with the path rules) over in the default namespace. I'm not exactly sure how but it seems to be working great! :)
I speculate maybe because the Ingresses are all defined with the same host that is the glue that helps wire things up - everything is currently working in the desired way with a single LB + services on a different namespace.

in my ingress-basic namespace:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ing
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  tls:
  - hosts:
    - example.dev
    secretName: tls-secret
  rules:
  - host: example.dev

I didn't define specific http.paths here in the 'parent' Ingress resource, since it feels more like a concern of the individual service, not the whole ingress.

Meanwhile over in the default namespace:
service1:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ing-be
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  tls:
  - hosts:
    - example.dev
    secretName: tls-secret
  rules:
  - host: example.dev
    http:
      paths:
      - path: /api(/|$)(.*)
        backend:
          serviceName: be-service
          servicePort: 80

service2:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ing-fe
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  tls:
  - hosts:
    - example.dev
    secretName: tls-secret
  rules:
  - host: example.dev
    http:
      paths:
      - path: /(.*)
        backend:
          serviceName: fe-service
          servicePort: 80

[cfecherolle]

+1 on this. In our case, we separate our environments in different namespaces, each one having its own Ingress.

We wanted to transparently redirect to a unique service instance (an IAM) used by all of these environments, by adding rules in each Ingress config, but it turns out ExternalName services are not accepted as targets so we can't.

I don't see any obvious solution: IAM service itself is in another namespace so we can't reference it from our Ingresses in other namespaces, ExternalName is not accepted so this trickery (changing our IAM access service from NodePort to ExternalName) is also a no-no, we are just stuck on this issue right now.

I read some messages about implementing a "bridge" service, referencing the target ClusterIp of a sevice in another namespace but does this actually work? Has anyone succeeded and if so, how? I'm puzzled.

[carlosjgp]

+1 on this. In our case, we separate our environments in different namespaces, each one having its own Ingress.

We wanted to transparently redirect to a unique service instance (an IAM) used by all of these environments, by adding rules in each Ingress config, but it turns out ExternalName services are not accepted as targets so we can't.

I don't see any obvious solution: IAM service itself is in another namespace so we can't reference it from our Ingresses in other namespaces, ExternalName is not accepted so this trickery (changing our IAM access service from NodePort to ExternalName) is also a no-no, we are just stuck on this issue right now.

I read some messages about implementing a "bridge" service, referencing the target ClusterIp of a sevice in another namespace but does this actually work? Has anyone succeeded and if so, how? I'm puzzled.

@cfecherolle You just need to deploy an ingress and service on a namespace where the service is configured as ExternalName to the service in the other namespace

[cfecherolle]

Hi @carlosjgp.
I'm not sure I understand your answer. Could you please clarify? Which namespace, which ingress and service, in short : how do you separate and wire things together, regarding namespaces?

[SpectralHiss]

I am not sure if the issue is satisfactorily resolved, in which cases does the externalname trick actually work?
to confirm for AWS EKS (v1.16) with alb ingress controller I don't seem to be able to have a service in namespace B (where ingress lives) point to another with the same name in namespace A.
I tried two approaches:

Using headless endpoint service

apiVersion: v1
kind: Service
metadata:
  namespace: NAMESPACEB
  name: checks-service-integration-service
  annotations:
    "alb.ingress.kubernetes.io/healthcheck-path": "/v1/health_check"
    "alb.ingress.kubernetes.io/target-type": "ip"
    "alb.ingress.kubernetes.io/backend-protocol": "HTTP"
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: v1
kind: Endpoints
metadata:
  name: checks-service-integration-service
  namespace: NAMESPACEB
subsets:
  - addresses:
      - ip: 172.20.156.11 <- this is the service's virtualIP in NAMESPACEA
    ports:
      - protocol: TCP
        port: 80

This solution doesn't work (times out) due to kube-proxy not supporting virtualIP (according to docs) and this works if i put the pod ip but obviously not an acceptable solution.

Using externalname service (as suggested in above response)

apiVersion: v1
kind: Service
metadata:
  namespace: *NAMESPACEB*
  name: checks-service-integration-service
  annotations:
    "alb.ingress.kubernetes.io/healthcheck-path": "/v1/health_check"
    "alb.ingress.kubernetes.io/backend-protocol": "HTTP"
   "alb.ingress.kubernetes.io/target-type": "ip"
spec:
  externalName: checks-service-integration-service.NAMESPACEA.svc.cluster.local
  ports:
    - port: 80

This causes connection refused errors when accesed checks-service-integration-service.NAMESPACEA.svc.cluster.local.
I am specifying IP mode because i don't think externalName works with NodePort and hence cannot use instance mode..

Thanks!

[rymnc]

@SpectralHiss were you able to get it to work?

[spstarr]

I would have thought this would have been supported, for blue-green setups using two namespaces...

[Dmitry1987]

I want to suggest a trick for anyone stuck trying to figure out how to route across namespaces, you can deploy an nginx pod or few of them, with config map that has upstream addresses defined manually if your namespaces have known names in advance. but if you're about to also scale namespaces (adding more and removing) then making this config map update itself with a script will be required, and also reload the config in nginxs (dispatch an 'exec' with reload command or smth like that). I'm about to try that but so far seems like the most straightforward solution because k8s internal resources do not support load balancing across namespaces so we have no choice but hack our way manually :) . it's a bad architecture design though, this should not really happen in production (we do it for a quick experiment, but such thing would never go to production lol, it's just awful design for microservices infra)

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-proxy-config
data:
  nginx.conf: |
    events {
      worker_connections  1024;
    }
    http {
      upstream backend {
        least_conn;
        server app1-service.namespace1.svc.cluster.local:80;
        server app2-service.namespace2.svc.cluster.local:80;
      }
      server {
        listen 80;

        location / {
          proxy_pass http://backend;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
      }
    }

[piradata]

On AWS I use a single network load balancer for a LOT of ingress and services in different name-spaces, one single nginx controller can handle multiple ingresses on different name-spaces normally

[bss-dmitry-shmakov]

On AWS I use a single network load balancer for a LOT of ingress and services in different name-spaces, one single nginx controller can handle multiple ingresses on different name-spaces normally

yes but the thread is about ingress rules ability to instruct the ingress-controller about where to route the requests following a rule. at the moment it's impossible to specify inside ingress rule a target service endpoint or name from another namespace (where this ingress rule does not reside). and ingress rules are namespace scoped resources (not global), so this kind of instruction is not possible in them (with no regards of what the ingress-controller can or cannot do :) I hope it clarifies what the thread was about, but thanks for chiming in, a single nginx and aws LBs definitely can be used for a lot of namespaces and very large clusters, that is correct).