[RFC] Canary deployments for cloudflare workers

[chronark]

Cloudflare workers don't have any blue-green or canary deployments out of the box, but it would be very useful for us to deploy safely by minimizing the blast radius of bad releases.

A potential solution for this would be building our own canary rollouts using GitHub actions, axiom and a canary worker.

The idea is that the current production worker is capable of redirecting a portion of traffic via service binding to a canary worker. the rollout is enabled and configured via environment variable:

• CANARY_ROLLOUT=10 would indicate 10% of the traffic is forwarded to the canary.
• CANARY_ROLLOUT=0 or not set, means no traffic is forwarded.

The canary worker can be either accessed via canary.unkey.dev or through a service worker binding, which would run the worker in the same process, thus not introducing any latency.

A push to main will trigger this workflow and a manual approval by the maintainers is required to proceed in the beginning.

A GitHub action will first deploy the new codebase to canary and then start the rollout by setting the environment variable in production.
The canary worker will log warnings and errors to axiom, which can then be queried by the GitHub Action and any error will stop the rollout and alert us in slack. If there are no errors or other performance regressions, we proceed.

These steps are repeated until we have rolled out to a specific percentage, see below.

Initially I'd use these numbers to configure the rollout, but we can adjust them.

1. 1% for 5min
2. 5% for 5 min
3. 10% for 5 min
4. full rollout by deploying to production

[RohitLakh]

• When you say the 10% of traffic, will that traffic distribution be random or based on some logic by user, workspace, free, or paid users?
• Doesn't it make sense to run this when a new release is created rather than a push to main branch, since that is the base branch that we merge all code changes to?
• What type of errors are we talking about in axiom and what is the percentage of degradation that will trigger the re-deploy of older code? Since not all errors are equal and need a redeploy.

[chronark]

When you say the 10% of traffic, will that traffic distribution be random or based on some logic by user, workspace, free, or paid users?

in the beginning it'll just be 10% random, but we can flag by api later, good idea

Doesn't it make sense to run this when a new release is created rather than a push to main branch, since that is the base branch that we merge all code changes to?

maybe, since these still need to be approved, it would be fine to trigger on merge

What type of errors are we talking about in axiom and what is the percentage of degradation that will trigger the re-deploy of older code? Since not all errors are equal and need a redeploy.

any error will trigger a rollback, if it's not critical or something that requires human attention, it would not be logged as error but as warning.

in addition: metrics such as latency, cache hits, or db calls could be good indicators that something went wrong and we should roll back

[RohitLakh]

in the beginning it'll just be 10% random, but we can flag by api later, good idea

I think better would be to flag it by users (Paid vs Free), so paid users are never affected by service breakdown.

maybe, since these still need to be approved, it would be fine to trigger on merge

From what I have seen some initiatives take multiple merges which might break prod, anyway, there is a reason we create releases (for releasing, it's an active process, whereas merging is a passive process) so it did make sense to have it on release but that's something that we can iterate over time.

any error will trigger a rollback, if it's not critical or something that requires human attention, it would not be logged as error but as warning.

How do we define something in code that axiom will know that it's a critical error? Database connection error or Internal Server Error then how many? Since we are serverless some errors might not be worth a redeploy so how do we define them?

[chronark]

logger.error(...) will log an error to axiom, that's all it takes

not sure what you imply by saying serverless is different. an error is an error :D

[RohitLakh]

Let's say database error, even if we are logging it by logger.error(...) that's still doesn't require a redeploy of a previous version but needs something else to be looked at.