So, you wanna hotfix, eh?

When critical bugs occur in production, or in a train that's being tested, and we decide to fix them, we call it a "hotfix". There's a process for this, and this page explains it.

Normal Hotfixes to rolling trains

When hotfixes are not related to security issues that could cause users harm if they were public, here's how it works:

1. Ensure a github issue is opened for the bug. The bug should include: 1. the clearly stated user visible manifestation of the issue 2. the estimated percentage of users affected 3. the estimated complexity/risk of the fix
2. write an email involving QA and ops, pointing at the issue
3. come to quorum on whether the bug should allowed to ship, should be hotfixed, or should derail the train
4. if a hotfix is desired, write the hotfix
5. open a pull request to merge the hotfix into the rolling train
6. have a colleague review the pull request, and merge it into the train
7. update the ChangeLog documenting your hotfix, i.e.: (hotfix 2012.04.03) Fixed foo: #1234
8. update the Release field in scripts/browserid.spec
9. push the above two things (probably together in one commit) up to the appropriate branch in our public repo
10. update the stage deployment ticket, mentioning the version, sha, and location where the code resides. Also include a brief one-line description of the differences between the previous version (what is the hotfix for?)
11. make sure ops knows to deploy, make sure QA knows to test, go have a beer.
12. Follow up with OPs to make sure HotFix is deployed. Follow up with QA to make sure the HotFix is tested and works as expected: "it solves the problem"
13. Repeat steps 10 - 12 for production.

Security Related Hotfixes to rolling trains

What if the hotfix addresses a security issue that could hurt users if it were made public before it were patched?

Well, use exactly the same steps as above, EXCEPT:

• For step #1 - use bugzilla and mark the bug as "private", instead of using github issues
• For step #9, use the private browserid repo
• for step #10, use the proper URL for the private repo

For Hotfixes for trains currently in production

Exactly the same steps as above, except you must open a new bugzilla ticket, and ops will first deploy to our staging environment (pausing all testing) so QA can vet the change before we take it live into production.