PROD Issue - Batch Job Issue: Published FOMs Did Not Get Switched To Commenting Open On Opening Date.

[ianliuwk1019]

Describe the Bug
Julius reported that a FOM in Published state but commenting opening date is today in production did not get changed to status to be Commenting Open.

Expected Behaviour
Status should be Commenting Open.

Actual Behaviour
Still on Published status.

** Steps To Reproduce**
See screenshot below (Production)

Screenshots

Additional context

• As of today FOM also havs pending changes in pipeline TEST.
• Additional related tickets: bug: reported no PROD-labeled images #561 , bug: API and cronjob using different images #562

[MCatherine1994]

Issue and Solution:

This issue is cased by the cronjob uses wrong image. As shared in #562, the cronjob pulls image from GitHub image registry, uses the latest "test" tag image. However, since we're doing some recent improvement and test deployment, the latest "test" tag image has all our new features. The cronjob argument got changed, so the checks to run the state change cronjob failed and it can't be triggered anymore.

And if we check the api deployment config, it's pulling the image from the Openshift image stream. It uses the latest "test" image when we did prod deployment last time, which is expected and correct.

So in the fix ticket #562, pr #568, we updated the api/openshift.deploy.yml, to let cronjob pull the image from Openshift image stream as well. The workflow works as below:

• Currently when we merge a pr, it will do test deployment (.github/workflows/merge-main.yml)
• And when the pr merged to main and closed, it will set "test" tag to the image (.github/workflows/pr-close.yml)
• When we do prod deployment, it will push the latest "test" tag image from GHCR to Openshift image stream
• And after prod deployment, it will set the "prod" tag to the latest "test" tag image we used for prod deployment
  So as long as both cronjob and api are pulling image from the Openshift image stream, our test deployment won't impact them anymore. We only push "test" tag image to Openshift image stream when doing prod deployment.

Deployment Condition
We did a PROD deployment on Wednesday to get all business required feature improvement in prod. And that make the cronjob argument works again to trigger the cronjob. So this problem is solved temporarily.

The complete fixes are done in pr #568, and it's merged to main and deployed in test now. We still need to deploy the fix to production to avoid getting the same issue in the future.