[dev] Double notifications from remind-to-promote bot

[tsibley]

The bot which reminds us to promote canary to production in a timely fashion has been sending us double notifications, some time apart.

2023-07-19 18:13 — Nextstrain Bot
  Changes are lingering on next.nextstrain.org.
  Review the changes and promote (if appropriate) via the Heroku Dashboard.

2023-07-20 00:50 — Nextstrain Bot
  Changes are lingering on next.nextstrain.org.
  Review the changes and promote (if appropriate) via the Heroku Dashboard.

These correspond to the following GitHub Actions workflow runs:

• https://github.com/nextstrain/nextstrain.org/actions/runs/5605716109/job/15186858976
• https://github.com/nextstrain/nextstrain.org/actions/runs/5608320463/job/15193697092

Both of those runs produced this state:

{
  "config": {
    "pipeline": "38f67fc7-d93c-40c6-a182-501da2f89d9d",
    "staging": "nextstrain-canary",
    "production": "nextstrain-server"
  },
  "staging": {
    "slug": "ae11b5d2-a798-489a-ada0-c86920e64ab9",
    "commit": "70c3b644d172898cf489113e77b6ac2510998a62",
    "releasedAt": "2023-07-14T22:03:29Z"
  },
  "production": {
    "slug": "8f5b7826-2502-45d1-bc48-30b865586377",
    "commit": "5b6808442908093f58bde9259916616a4abf1062"
  }
}

which hashes to:

61d84dae9d18e7a88f726875037d7675b246b8a99a086f68a15cf24759c93476

The caching behaviour of the two runs was:

restored from remind-to-promote-state-hash-5605206714
stored to remind-to-promote-state-hash-5605716109

restored from remind-to-promote-state-hash-5607826181
stored to remind-to-promote-state-hash-5608320463

So why did it notify twice if the contents and hash hadn't changed? Because of the intervening runs meant the cached state did change. Notice that the second run (obviously in hindsight) doesn't restore from the state stored by the first run, but from the immediately prior run. That run is:

• https://github.com/nextstrain/nextstrain.org/actions/runs/5607826181/job/15192288310

It produced an empty state file (because No one's around; skipping check.) hashing to:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

And therein lies the issue. The state is always chained from and compared to the immediately prior run. The double notification bug arises because we're overloading the state's non-existence to mean both "nothing to promote" and "something to promote, but not yet".