Hybrid Cloud

Download Sources

Lets use the following variables to refer to the source repositories :

• $HYBRID_CLOUD_HOME — https://github.com/redhat-developer-demos/hybrid-cloud-serverless

• $BACKEND_APP_HOME — https://github.com/redhat-developer-demos/hybrid-cloud-app-backend

• $FRONTEND_APP_HOME — https://github.com/redhat-developer-demos/hybrid-cloud-app-frontend

git clone https://github.com/redhat-developer-demos/hybrid-cloud-serverless
export HYBRID_CLOUD_HOME=`pwd`/hybrid-cloud-serverless
git clone https://github.com/redhat-developer-demos/hybrid-cloud-app-backend
export BACKEND_APP_HOME=`pwd`/hybrid-cloud-app-backend
git clone https://github.com/redhat-developer-demos/hybrid-cloud-app-frontend
export FRONTEND_APP_HOME=`pwd`/hybrid-cloud-app-frontend

Prerequisites

• OpenShift4 cluster on three clouds with following components installed:

  • OpenShift Serverless

• skupper

• yq

Let us call the OpenShift Clusters as Cloud-1(gcp), Cloud-2(azr) and Cloud-3(aws).

The following table shows the cloud and what components gets installed on each of them:

Table 1. Clouds and Components

Component	Cloud-1	Cloud-2	Cloud-3
Cloud Provider	gcp	azr	aws
Backend	✓	✓	✓
Frontend	✓	❌	❌
Generate Site Token(token.yaml)	✓	❌	❌
Weight	0	15	20

Note	You can use any cloud provider for any OpenShift4 supported cloud, the gcp/aws/azr is used in this example. Weight controls how many requests that cloud can handle before skupper bursts them out to other clouds

Before the backend or frontend applications are deployed, execute the following on each cloud:

cd $HYBRID_CLOUD_HOME
oc apply -k k8s/core
oc project hybrid-cloud-demo
oc apply -k k8s/rbac

Cloud-1

Add Image Streams

oc import-image hybrid-cloud-backend --confirm --all --from quay.io/rhdevelopers/hybrid-cloud-demo-backend
# set local lookup policy
oc set image-lookup hybrid-cloud-backend
oc import-image hybrid-cloud-frontend --confirm --all --from quay.io/rhdevelopers/hybrid-cloud-demo-frontend
# set local lookup policy
oc set image-lookup hybrid-cloud-frontend

oc apply -k k8s/skupper/gcp

Cloud-2

Add Image Streams

oc import-image hybrid-cloud-backend --confirm --all --from quay.io/rhdevelopers/hybrid-cloud-demo-backend
# set local lookup policy
oc set image-lookup hybrid-cloud-backend

oc apply -k k8s/skupper/azr

Cloud-3

Add Image Streams

oc import-image hybrid-cloud-backend --confirm --all --from quay.io/rhdevelopers/hybrid-cloud-demo-backend
# set local lookup policy
oc set image-lookup hybrid-cloud-backend

oc apply -k k8s/skupper/aws

Run the following commands on Cloud-1, Cloud-2 and Cloud-3 to wait for skupper deployments to be ready:

oc rollout status deployment skupper-site-controller
oc rollout status deployment skupper-router
oc rollout status deployment skupper-service-controller

Backend Deployment

On all clouds Cloud-1, Cloud-2 and Cloud-3, run the following commands to deploy Knative backend:

Deploy backend application(Knative):

cd $BACKEND_APP_HOME
oc apply -k k8s/knative

Run the following command to check the status:

oc get pods,svc,ksvc

A successful deployments of core,rbac,skupper and backend components, should show an output like:

NAME                                                        READY   STATUS    RESTARTS   AGE
pod/hybrid-cloud-backend-p948k-deployment-b49c9569b-ggv8z   2/2     Running   0          26s
pod/skupper-router-56c4544bbc-dhckt                         3/3     Running   0          43m
pod/skupper-service-controller-5bcf486799-v2hl2             2/2     Running   0          43m
pod/skupper-site-controller-5cf967f858-z2dx8                1/1     Running   0          43m

NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                  PORT(S)                             AGE
service/hybrid-cloud-backend                 ExternalName   <none>           kourier-internal.knative-serving-ingress.svc.cluster.local   <none>                              21s
service/hybrid-cloud-backend-p948k           ClusterIP      172.30.223.229   <none>                                                       80/TCP                              26s
service/hybrid-cloud-backend-p948k-private   ClusterIP      172.30.140.107   <none>                                                       80/TCP,9090/TCP,9091/TCP,8022/TCP   26s
service/hybrid-cloud-backend-skupper         LoadBalancer   172.30.1.23      <pending>                                                    80:31554/TCP                        29s
service/skupper-controller                   ClusterIP      172.30.119.15    <none>                                                       443/TCP                             43m
service/skupper-internal                     ClusterIP      172.30.205.136   <none>                                                       55671/TCP,45671/TCP                 43m
service/skupper-messaging                    ClusterIP      172.30.14.214    <none>                                                       5671/TCP                            43m
service/skupper-router-console               ClusterIP      172.30.72.116    <none>                                                       443/TCP                             43m

NAME                                               URL                                                                 LATESTCREATED                LATESTREADY                  READY   REASON
service.serving.knative.dev/hybrid-cloud-backend    http://hybrid-cloud-backend.hybrid-cloud-demo.svc.cluster.local   hybrid-cloud-backend-p948k   hybrid-cloud-backend-p948k   True

Connecting Clouds

On Cloud-1, run the following command to create site-token secret:

cd $HYBRID_CLOUD_HOME
oc apply -k k8s/van

The site-token seceret will be used to connect clouds Cloud-2 and Cloud-3 to Cloud-1 forming a Virtual Application Network(VAN).

Run the following command to export the site-token secret:

oc get secret -n hybrid-cloud-demo site-token -o yaml > $HYBRID_CLOUD_HOME/token.yaml

To connect Cloud-2 and Cloud-3 with Cloud-1, run the following command on Cloud-2 and Cloud-3:

Cloud-2

Make sure the Cloud-2 starts to spill over after 4 requests in queue:

yq w $HYBRID_CLOUD_HOME/token.yaml 'metadata.annotations[skupper.io/cost]' --tag '!!str' 15 | oc create -n hybrid-cloud-demo -f -

Cloud-3

Make sure the Cloud-3 starts to spill over after 2 requests in queue:

yq w $HYBRID_CLOUD_HOME/token.yaml 'metadata.annotations[skupper.io/cost]' --tag '!!str' 20 | oc create -n hybrid-cloud-demo -f -

A successfull connection can be verified using the following commands, on Cloud-1:

export SKUPPER_POD=$(oc get pods --selector=application=skupper-router -ojsonpath='{.items[0].metadata.name}')
kubectl exec -it $SKUPPER_POD -- qdmanage query --type node

The command should show an output as shown below:

[
  {
    "index": 0,
    "nextHop": "(self)",
    "name": "router.node/hybrid-cloud-gcp-skupper-router-7b9fcf9575-knrsd",
    "validOrigins": [],
    "linkState": [
      "hybrid-cloud-azr-skupper-router-675cb597fc-75g2x",
      "hybrid-cloud-aws-skupper-router-5946d4544f-mgswf"
    ],
    "instance": 1599839762,
    "identity": "router.node/hybrid-cloud-gcp-skupper-router-7b9fcf9575-knrsd",
    "protocolVersion": 1,
    "lastTopoChange": 1599840055,
    "type": "org.apache.qpid.dispatch.router.node",
    "id": "hybrid-cloud-gcp-skupper-router-7b9fcf9575-knrsd",
    "address": "amqp:/_topo/0/hybrid-cloud-gcp-skupper-router-7b9fcf9575-knrsd"
  },
  {
    "routerLink": 0,
    "name": "router.node/hybrid-cloud-aws-skupper-router-5946d4544f-mgswf",
    "index": 1,
    "validOrigins": [
      "hybrid-cloud-azr-skupper-router-675cb597fc-75g2x"
    ],
    "protocolVersion": 1,
    "linkState": [
      "hybrid-cloud-gcp-skupper-router-7b9fcf9575-knrsd"
    ],
    "instance": 1599840049,
    "cost": 15,
    "address": "amqp:/_topo/0/hybrid-cloud-aws-skupper-router-5946d4544f-mgswf",
    "type": "org.apache.qpid.dispatch.router.node",
    "id": "hybrid-cloud-aws-skupper-router-5946d4544f-mgswf",
    "identity": "router.node/hybrid-cloud-aws-skupper-router-5946d4544f-mgswf"
  },
  {
    "routerLink": 1,
    "name": "router.node/hybrid-cloud-azr-skupper-router-675cb597fc-75g2x",
    "index": 2,
    "validOrigins": [
      "hybrid-cloud-aws-skupper-router-5946d4544f-mgswf"
    ],
    "protocolVersion": 1,
    "linkState": [
      "hybrid-cloud-gcp-skupper-router-7b9fcf9575-knrsd"
    ],
    "instance": 1599840051,
    "cost": 20,
    "address": "amqp:/_topo/0/hybrid-cloud-azr-skupper-router-675cb597fc-75g2x",
    "type": "org.apache.qpid.dispatch.router.node",
    "id": "hybrid-cloud-azr-skupper-router-675cb597fc-75g2x",
    "identity": "router.node/hybrid-cloud-azr-skupper-router-675cb597fc-75g2x"
  }
]

Verify Status

Cloud-1

Running skupper status on the clouds should show the following output:

Skupper is enabled for namespace '"hybrid-cloud-demo" in interior mode'. It is connected to 2 other sites. It has 1 exposed service.

Cloud-2

Running skupper status on the clouds should show the following output:

Skupper is enabled for namespace '"hybrid-cloud-demo" in interior mode'. It is connected to 2 other sites (1 indirectly). It has 1 exposed service.

Cloud-3

Running skupper status on the clouds should show the following output:

Skupper is enabled for namespace '"hybrid-cloud-demo" in interior mode'. It is connected to 2 other sites (1 indirectly). It has 1 exposed service.

Note	Since Cloud-1 is where we generated the site-token secret to connect to other sites, makes it directly connected to other clouds Cloud-2 and Cloud-3. For other clouds you will see one direct to Cloud-1 and one indirect connection to other cloud.

Verify Exposed Services

You can verify that, in all connected clouds running the following command skupper list-exposed shows the following output:

When Knative backend was deployed:

Services exposed through Skupper:
    hybrid-cloud-backend-skupper (http port 80) with targets
      => hybrid-cloud-backend.hybrid-cloud-demo name=hybrid-cloud-backend.hybrid-cloud-demo

When standard Kubernetes (vanilla) backend was deployed:

Services exposed through Skupper:
    hybrid-cloud-backend (http port 80) with targets
      => hybrid-cloud-backend.hybrid-cloud-demo name=hybrid-cloud-backend.hybrid-cloud-demo

Frontend Deployment

On Cloud-1 deploy the frontend by running the following command:

cd $FRONTEND_APP_HOME
oc apply -k k8s/knative

Get the URL to access the frontend application:

export API_URL=http://$(oc get route -n hybrid-cloud-demo hybrid-cloud-frontend -ojsonpath='{.spec.host}')

Burst Testing

It is possible to verify the brust without user input using the following hey scripts:

Cloud-1 burst to Cloud-2

In order to burst from Cloud-1 to Cloud-2, you need to send atleast 15 requests to the API:

hey -z 2s -c 20 -m POST -d '{"text": "1+2","uppercase": false,"reverse": false}' -H "Content-Type: application/json" $API_URL/api/send-request

Cloud-1 burst to Cloud-2 burst to Cloud-3

In order to burst from Cloud-1 to Cloud-2, you need to send atleast 15 + 20 = 35 requests to the API:

hey -z 2s -c 35 -m POST -d '{"text": "1+2+3","uppercase": false,"reverse": false}' -H "Content-Type: application/json" $API_URL/api/send-request