Cloud Isolation - For separation of Lighthouse/vets-api, utilize ALB only, generic reverse proxy, or API Gateway,

[dginther]

• Start Date: 2022-03-24
• Relevant Issue: Create RFC for ALB / API Gateway / Nginx decision making #39028

Summary

We intend to separate the Lighthouse API (and give control of the associated domains *-api.va.gov and api.va.gov to Lighthouse) from the APIs hosted in the vets-api process, as well as any other internal API services that are hosted within the DSVA AWS account.

To this end, we need to determine what will route the traffic to vets-api and any other internal API services hosted in the DSVA account.

Motivation

In order to ensure proper routing of traffic to the vets-api (and any other services, or future services) APIs, we will need something to replace the functionality that is currently controlled by a combination of the revproxy and/or Kong.

In addition, if a team wanted to bring up an API that was not inside of vets-api, it would be nice to be able to offer an easy path to integrating that API into the existing set of APIs without a lot of hassle.

Possible Options

Currently, the Lighthouse team has a Kong (modified nginx) server which routes requests which are routed from the revproxy to the lighthouse APIs. Some of these requests appear to eventually make it back to the vets-api hosted services as well, we are investigating the Kong routes currently.

Once the Lighthouse team has transitioned to the Google Apigee software, they will no longer need (or want!) to maintain the Kong instances in the DSVA DVP subnets. This presents the platform with a decision to be made.

It is possible that we could just use some routing rules on an AWS ALB to send the traffic to the correct instances (and in the future, to the EKS cluster) This supposes that there is no need for manipulation of the traffic between the time it enters from the endpoint and when it reaches the internal endpoints (vets-api). It is my assumption that this will not be technically feasible and that some manipulation will need to be done.

If it is determined that the plain ALB is not a sufficient solution, we are presented with various options. All of these options initially take ingress traffic from an AWS ALB.

1. Use the revproxy we already have to route the traffic.

• This is an "Openresty" reverse proxy (based on nginx)
• We currently configure this reverse proxy with BRD, and once we move the revproxy into EKS, it will be configured by a jinja2 template init container (https://github.com/department-of-veterans-affairs/vsp-jinja2-init)
• To my knowledge, it is intended to eventually replace this Openresty server with a collection of rules in the EKS ingress controller, "Traefik"
• The configuration of the Openresty server via templates is a complicated and somewhat hard process to understand, it has a learning curve, because the templates are complicated with a lot of conditional statements.
• Makes addition of new services, especially ones not encompassed inside vets-api more difficult, and require more configuration.

2. Use the already installed EKS Ingress controller, "Traefik", along with configured middlewares to route traffic to APIs.

• We currently use the 'Community' version of Traefik as an ingress controller on the EKS clusters.
  • There are middlewares available for manipulating and mangling ingress traffic and headers
  • There is a Traefik 'Enterprise' version available (for a cost) which would enable more specific API capabilities
• Configuration is dynamic, Kubernetes yaml resources are used to control the rules and middlewares
• Not as fully featured as a purpose-built API Gateway software
• Would make integration of future services moderately easier

3. Use a purpose-built API Gateway software.

• There are a variety of API Gateway products, prefer OSS
  • Possible options include things like:
    • Tyk (or Tyk Operator)
    • Fusio
    • Envoy-based solutions
    • Kong
• Would require a new product to be spun up likely in the EKS infrastructure
• Could make future integration of new services extremely simple

4. Inherit the Kong instances from Lighthouse

• Not recommended

[kylesoskin]

It is possible that we could just use some routing rules on an AWS ALB to send the traffic to the correct instances (and in the future, to the EKS cluster) This supposes that there is no need for manipulation of the traffic between the time it enters from the endpoint and when it reaches the internal endpoints (vets-api).

Doesn't API gateway easily allow for transforms/manipulation of traffic going through the aws gateway (either via a aws gateway integration, or a lambda?). Is there any oppurtunity to use AWS Gateway instead of just AWS ALB if traffic needs adjusted?

This could make routing everything easier since it is all going to AWS stuff (EKS or other entities living in AWS)

[dginther]

Yes, AWS API Gateway could maybe be used in this way, but my initial perusal of their documentation leaves me wondering if it is a service that we would be able to easily manage.
From their docs:

Are HTTP-based.

Enable stateless client-server communication.

Implement standard HTTP methods such as GET, POST, PUT, PATCH, and DELETE.

and then https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html#getting-started-create-api

It appears that each "API" that you create is separately managed, and intended to hit a back-end service like Lambda or some other service, but that each one of these separate APIs would then be it's own discrete API, not combined behind a single URL or have the ability to manipulate the traffic in any way. I think from my reading of the docs that this would still put us in a position of having multiple backend API addresses and still having no good way of combining them into a unified single API.

I see that they have a concept of OpenAPI extensions as well, which could be used to manipulate the traffic as needed.

I could be misunderstanding the intention of the service, but it looks, like with many other AWS services, like a user-unfriendly, half-baked product that might be good in 5 years, after they have paid some attention to it.

[kylesoskin]

I used the API Gateway a few years ago and it was good for "making stuff happen" in AWS, but for just guiding traffic, I agree it would/could be too complicated and not needed.

I have used it before to do things like:

external app -> aws gateway --[pass payload to lambda]--> lambda transforms payload -> AWS SQS/S3

or

external app -> aws gateway -> lambda that spins up or down auto-scaling ec2 group

If it ONLY needs to route, yea it will be overkill. But if there is routing and transforming needing to happen, it may not be.
If more AWS services were being used it may be more useful, but for what it sounds like, yea it's probably overkill.

[dginther]

I was thinking something like this: https://tyk.io/open-source-api-gateway/

[dsteeve]

Once the Lighthouse team has transitioned to the Google Apigee software, they will no longer need (or want!) to maintain the Kong instances in the DSVA DVP subnets. This presents the platform with a decision to be made.

Is this statement true? The last I heard this was being discussed on how to go about it. Maybe this RFC IS that discussion?

[madebydna]

We recently met with the Lighthouse team and they confirmed that part of their plan of moving to GCP and Apigee is to discontinue the use of Kong as an API gateway in the DSVA account. We actually have a document that summarizes their staged migration plan and is based on architecture diagrams that they shared with out team. Stage 4 of the plan shows the absence of Kong in the DSVA account.

[timwright12]

No comments, I'm just dropping this link in case folks aren't familiar with AWS ALB (I had to Google it): https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html

[dginther]

Moved to #39990

[mchelen-gov]

RFCs have been moved to: https://github.com/department-of-veterans-affairs/va.gov-platform-architecture
Please migrate content there.