production: Deploy the bot and worker stack in openshift cluster

[vishnoianil]

Currently we are maintaining aws worker machines for running the precheck/generate jobs (through the remote model endpoints). Deploying the stack on openshift can give us all the goodness of scaling the bot stack. We can possibly run the local generate and serve the model if the cluster provides GPU.

[Gregory-Pereira]

Thinking of the Rosa ET cluster for this with Dave's changes having landed. Thoughts?

[vishnoianil]

Yeah, that's what i was thinking as well. Currently the worker node's IP are explicitly allowed to access the backend sdg service, we need to resolve that issue first to deploy it in the cluster to make it work out of the box.