No-downtime deploy

[ryanhugh]

It would be pretty cool if we could deploy with no downtime. Currently, there is 1 to 2 seconds of downtime every time the prod branch is pushed. I am thinking that one way we could do this is have two instances of the searchneu backend running on the EC2 server and listening on different ports. Nginx would send requests to both of these instances. Then, when we are updating the server, we can take update these instances one at a time and the the site will be up the entire time.

[ryanhugh]

Now that the backend is ElasticSearch, this could be easier. Give this a read!

https://www.elastic.co/blog/changing-mapping-with-zero-downtime

[dajinchu]

Yeah with elasticsearch we should also make use of index aliases: reindex everything to a new index, then swap the alias to point at the new index, which is an atomic operation