Rolling upgrades for major releases with no downtime

[teolupus]

People running ES at scale for critical applications cannot afford downtime for major upgrades. We all know ES would love to and has aspired to introduce this feature for a long time.

With technologies like Solrcloud supporting rolling upgrades (as per https://cwiki.apache.org/confluence/display/solr/Upgrading+a+Solr+Cluster), it is getting harder to justify using ES to back up critical search infrastructure. No question it is the preferred technology for developers, but it also needs to support the engineers that sweat day and night to keep their ES cluster's lights on.

Could you share with us if the rolling upgrade feature is on the roadmap and for what release of ES it is planned for?

[s1monw]

@teolupus thanks for opening this issue. It's a tough topic. I think I can say that we all try to get as much compatibility as possible to the user and reduce upgrade pain is a major topic in the dev team. From a engineering perspective I personally think that we just haven't been mature enough to do this from a products perspective. Full cluster restarts allow us to make changes that would not be possible otherwise like the two phase clusterstate commit since they require major changes. Doing this across major versions would be a huge hassle or rather too fragile for us to commit to it.

That said, we have come a long way since version 1.0 and the product as well as the testing has matured to discuss this topic again. By no means I want to commit to anything in this post but I'd want to start the discussion of how we could get to this point. I personally think we can't allow upgrades from N.X to N+1.Y ever, this must be something that is well defined and contained. Now, we are aiming for major releases every 8 to 12 months (if you plan for 8 it will be 12 :) ), that means we would need to keep wire protocol stable for 2 years.

Now one scenario that I could imagine is to narrow down the scope of the compatibility ie. if 5.3 is the last release in the 5.x series we allow rolling upgrades from 5.3 to 6.x only. This would help a lot with testing as well as allows us to specifically release bugfixes that cause problems along the lines. I think this could be a starting point for a discussion around this limitation.
I am curious what others think, I know there are people that will object or are critical but it's good to start writing them down.

[jasontedor]

With technologies like Solrcloud supporting rolling upgrades (as per https://cwiki.apache.org/confluence/display/solr/Upgrading+a+Solr+Cluster)

Solr does not support rolling upgrades across major versions. Please see this mail thread for reference.

[nik9000]

Now one scenario that I could imagine is to narrow down the scope of the compatibility ie. if 5.3 is the last release in the 5.x series we allow rolling upgrades from 5.3 to 6.x only.

I don't know that we'll be able to get it for 5.latest -> 6.x but its a good idea. I like limiting the scope of things we have to test.

[teolupus]

If ES supports rolling upgrades for major releases, I'd expect and agree it is a requirement to be on the latest minor release before I can do it.

I appreciate Elastic wants to be sure it can introduce ground breaking changes to keep evolving the product as fast as possible. That said, this needs to be balanced with the ability for big customers to continue to be big customers and roll their clusters on to the next version.

I wonder if this could be initially tested as a soft commitment rather then a rule. Ie: where possible we will allow you to do a rolling upgrades from latest minor to next major, but we do reserve the right to require downtime to introduce ground breaking features that our users need.

Maybe, after running this as an experiment for a while, we will all be in a better place to discuss this topic again.

[javanna]

Elasticsearch 6.0.0-alpha1 is out and it supports rolling upgrades from Elasticsearch 5.latest. Same will be valid once Elasticsearch 6.0 GA is out. See https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released . Happily closing this issue :)

[teolupus]

This is great news and a major step forward for ES as a technology and business. I'm sure a lot of people in the community must be celebrating this right now. Congratulations and a big thanks to the people involved in this work.