feat(backend): Support running behind a corporate proxy [RHIDP-2217]

[rm3l]

Description

This PR allows running the Showcase app behind a corporate proxy, which was the root cause of a login failure using the GitHub auth behind a proxy.

It does so per the upstream instructions provided on this page: https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/help-im-behind-a-corporate-proxy.md
This way, it can honor the conventional HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables (and their lowercase equivalents).

The undici and global-agent packages added here allow covering settings for the native fetch and node-fetch respectively. Note that even if node-fetch is recommended for HTTP data fetching by Backstage packages [ADR013], we identified a couple of plugins making use of other libraries like Axios (like the Keycloak backend plugin - see RHIDP-2217 for the full list). Axios reportedly already honors the proxy environment variables above [doc].
I tested the Keycloak backend plugin as an example to ensure this.

Which issue(s) does this PR fix

• Fixes RHIDP-2217

PR acceptance criteria

Please make sure that the following steps are complete:

• GitHub Actions are completed and successful
• Unit Tests are updated and passing
• E2E Tests are updated and passing
• Documentation is updated if necessary (requirement for new features)
• Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

Local Development

You can run locally with yarn:

1. First start a proxy server - example with a Squid image:

podman container run --rm --name squid-container -e TZ=UTC -p 3128:3128 -it docker.io/ubuntu/squid:latest 

2. In a separate terminal, start the application with the proxy environment variables.

GLOBAL_AGENT_ENVIRONMENT_VARIABLE_NAMESPACE='' \
  HTTP_PROXY=http://localhost:3128 \
  HTTPS_PROXY=http://localhost:3128 \
  NO_PROXY='localhost,example.org' \
  yarn start

3. Inspect the proxy logs and you should see that requests are sent through the proxy, e.g.:

1715067448.970    185 10.42.0.19 TCP_TUNNEL/200 5171 CONNECT raw.githubusercontent.com:443 - HIER_DIRECT/185.199.111.133 -
1715067450.517    256 10.42.0.19 TCP_TUNNEL/200 5129 CONNECT raw.githubusercontent.com:443 - HIER_DIRECT/185.199.111.133 -

Full test

For a more complete test, the most challenging part of testing the changes here is to set up an environment where an application is forbidden access to the public Internet except through a given proxy.
We can simulate such an environment in a Kubernetes namespace with the help of Network Policies to control ingress/egress traffic.
For example:

1. Make sure the network plugin in your Kubernetes cluster supports network policies. I created a cluster with k3d (k3d cluster create), and it supports Network Policies out of the box.

2. Create a separate proxy namespace, and deploy a Squid-based proxy application there:

kubectl create namespace proxy

cat <<EOF | kubectl -n proxy apply -f -
apiVersion: v1
kind: Service
metadata:
  name: squid-service
  namespace: proxy
  labels:
    app: squid
spec:
  ports:
  - port: 3128
  selector:
    app: squid

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: squid-deployment
  namespace: proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "squid"
  template:
    metadata:
      labels:
        app: squid
    spec:
      containers:
      - name: squid
        image: docker.io/ubuntu/squid:latest
        ports:
        - containerPort: 3128
          name: squid
          protocol: TCP
EOF

6. Create the namespace where the Showcase application will be running

kubectl create namespace my-ns

7. Add the network policies in the namespace above. The first one denies all egress traffic except to the DNS resolver and the Squid proxy. The second one allows ingress and egress traffic in the same namespace, because the Showcase app pod needs to contact the local Database pod.

cat <<EOF | kubectl -n my-ns apply -f -
---
# Deny all egress traffic in this namespace => proxy settings can be used to overcome this.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-egress-with-exceptions
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  # allow DNS resolution (we need this allowed, otherwise we won't be able to resolve the DNS name of the Squid proxy service)
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
      - port: 53
        protocol: UDP
      - port: 53
        protocol: TCP
  # allow traffic to Squid proxy
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: proxy
    ports:
    - port: 3128
      protocol: TCP

---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-same-namespace
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}
  egress:
  - to:
    - podSelector: {}
EOF

8. Deploy the showcase application using the image from this PR (quay.io/janus-idp/backstage-showcase:pr-1225). I tested this using the RHDH Operator, but it should work with the Helm Chart as well. For example, to only test the GitHub login issue:
   a. Follow the instructions to create a GitHub app
   b. Make sure to use the image from this PR (spec.application.image field if using the Operator).
   c. GitHub login should fail unless the HTTP(S)_PROXY environment variables are set, like so:

# Make sure to create the appropriate ConfigMaps and Secrets
apiVersion: rhdh.redhat.com/v1alpha1
kind: Backstage
metadata:
  name: my-rhdh
spec:
  application:
    image: quay.io/janus-idp/backstage-showcase:pr-1225
    appConfig:
      configMaps:
        - name: app-config-rhdh
    dynamicPluginsConfigMapName: dynamic-plugins-rhdh
    extraEnvs:
      envs:
        - name: HTTP_PROXY
          value: 'http://squid-service.proxy.svc.cluster.local:3128'
        - name: HTTPS_PROXY
          value: 'http://squid-service.proxy.svc.cluster.local:3128'
        - name: NO_PROXY
          value: 'localhost'
        - name: ROARR_LOG
          # Logs from global-agent (to inspect proxy settings)
          value: 'true'
      secrets:
        - name: secrets-rhdh

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-rhdh
data:
  "app-config-rhdh-auth.yaml": |
    auth:
      environment: development
      session:
        secret: my-super-secret-secret
      providers:
        github:
          development:
            clientId: '${GH_CLIENT_ID}'
            clientSecret: '${GH_CLIENT_SECRET}'

  "app-config-rhdh-catalog.yaml": |
    catalog:
      import:
        entityFilename: catalog-info.yaml
        pullRequestBranchName: backstage-integration
      rules:
        - allow: [Component, System, Group, Resource, Location, Template, API]
      locations:
        # Note: integrations.github[].apps must be correctly configured to read GitHub locations
        - type: url
          target: https://github.com/janus-idp/backstage-showcase/blob/main/catalog-entities/all.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: dynamic-plugins-rhdh
data:
  dynamic-plugins.yaml: |
    includes:
      - dynamic-plugins.default.yaml
    plugins:
      - package: './dynamic-plugins/dist/backstage-plugin-catalog-backend-module-github-dynamic'
        disabled: false
        pluginConfig:
          catalog:
            providers:
              github:
                myorg:
                  organization: '${GH_ORG}'
                  schedule:
                    # supports cron, ISO duration, "human duration" (used below)
                    frequency: { minutes: 30}
                    # supports ISO duration, "human duration (used below)
                    timeout: { minutes: 3}
                    initialDelay: { seconds: 15}

---
apiVersion: v1
kind: Secret
metadata:
  name: secrets-rhdh
stringData:
  # generated with the command below (from https://backstage.io/docs/auth/service-to-service-auth/#setup):
  # node -p 'require("crypto").randomBytes(24).toString("base64")'
  BACKEND_SECRET: "R2FxRVNrcmwzYzhhN3l0V1VRcnQ3L1pLT09WaVhDNUEK" # notsecret
  GH_ORG: "my-gh-org"
  GH_CLIENT_ID: "changeme"
  GH_CLIENT_SECRET: "changeme"

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[gazarenkov]

Any tests expected?

[rm3l]

Any tests expected?

This will make sense as E2E tests, which are going to be covered in https://issues.redhat.com/browse/RHIDP-1955.

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[coreydaley]

@rm3l I think that the Local development and Full Test instructions that you wrote in the description for this pull request would be a great addition to a docs/proxy.md file as part of this pull request.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.1% Duplication on New Code

See analysis details on SonarCloud

[rm3l]

Rebased and force-pushed to fix a conflict with the following files:

• .rhdh/docker/Dockerfile
• docker/Dockerfile

[rm3l]

/cc @gashcrumb

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1225!

[rm3l]

/test e2e-tests

[kim-tsao]

/approve

[kim-tsao]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kim-tsao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kim-tsao]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment