DCOS_OSS-5277: Explore switching to ZooKeeper 3.5.x for DC/OS 1.14 #5878

jgehrcke · 2019-07-18T09:22:04Z

The first step is to see if integration tests pass.

https://jira.mesosphere.com/browse/DCOS_OSS-5277

d2iq-mergebot · 2019-07-18T09:22:07Z

This repo has @mesosphere-mergebot integration. You can perform the following commands by submitting a comment. Submit a comment with content "@mesosphere-mergebot help" to view more detailed help text and examples. Be sure the have a look at the mergebot documentation, too.

@mesosphere-mergebot changelog-not-required reason 
@mesosphere-mergebot sync  
@mesosphere-mergebot bump-ee  
@mesosphere-mergebot merge-it  
@mesosphere-mergebot override-status pr-status-check jira-url 
@mesosphere-mergebot request-review  
@mesosphere-mergebot label [Ship It|Request For Comment|Ready For Review|Holding|Work In Progress|merge-by-mergebot]

PR creators can apply one of [Ready For Review|Work In Progress]. Owners can apply any label.

d2iq-mergebot · 2019-07-18T09:22:07Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T09:22:35Z

Enterprise Bump PR: mesosphere/dcos-enterprise/pull/6194

d2iq-mergebot · 2019-07-18T09:59:30Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T09:59:56Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

d2iq-mergebot · 2019-07-18T10:12:40Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T10:13:06Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

jgehrcke · 2019-07-18T10:45:52Z

@mesosphere-mergebot override-status teamcity/dcos/test/terraform/aws/onprem/static/group3 https://jira.mesosphere.com/browse/DCOS-56126

d2iq-mergebot · 2019-07-18T10:45:54Z

@jgehrcke, only listed owners other than the PR creator may issue that command.

adamtheturtle · 2019-07-18T10:50:04Z

@mesosphere-mergebot override-status teamcity/dcos/test/terraform/aws/onprem/static/group3 https://jira.mesosphere.com/browse/DCOS-56126

adamtheturtle · 2019-07-18T10:50:21Z

(as discussed, @jgehrcke will re-run the overridden command).

d2iq-mergebot · 2019-07-18T11:39:07Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T11:39:33Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

d2iq-mergebot · 2019-07-18T12:59:55Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T13:00:22Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

jgehrcke · 2019-07-18T13:23:37Z

Got ZooKeeper to start, but Exhibitor does not detect that ZooKeeper is working. Changed log level output to DEBUG for both ZK and Exhibitor and I think this shows an important clue:

Jul 18 13:21:59 dcos-e2e-default-f52b5-master-0 java[5263]: [myid:1] INFO  [main:ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
Jul 18 13:21:59 dcos-e2e-default-f52b5-master-0 start_exhibitor.py[5149]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Starting zookeeper ... STARTED [pool-2-thread-2]
Jul 18 13:22:00 dcos-e2e-default-f52b5-master-0 java[5263]: [myid:1] DEBUG [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:53942
Jul 18 13:22:00 dcos-e2e-default-f52b5-master-0 java[5263]: [myid:1] INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 168541732
Jul 18 13:22:00 dcos-e2e-default-f52b5-master-0 java[5263]: [myid:1] INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[srvr]]
Jul 18 13:22:00 dcos-e2e-default-f52b5-master-0 java[5263]: [myid:1] DEBUG [NIOWorkerThread-1:NIOServerCnxn@511] - Command ruok is not executed because it is not in the whitelist.
Jul 18 13:22:00 dcos-e2e-default-f52b5-master-0 java[5263]: [myid:1] DEBUG [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:53942 (no session established for client)
Jul 18 13:22:00 dcos-e2e-default-f52b5-master-0 start_exhibitor.py[5149]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper down/not-serving waiting 2021 of 40000 ms before restarting [ActivityQueue-0]

jgehrcke · 2019-07-18T13:26:05Z

Command ruok is not executed because it is not in the whitelist.

https://issues.apache.org/jira/browse/ZOOKEEPER-2764

4lw.commands.whitelist
(Java system property: zookeeper.4lw.commands.whitelist)

New in 3.5.3: A list of comma separated Four Letter Words commands that user wants to use. A valid Four Letter Words command must be put in this list else ZooKeeper server will not enable the command. By default the whitelist only contains "srvr" command which zkServer.sh uses. The rest of four letter word commands are disabled by default.

Here's an example of the configuration that enables stat, ruok, conf, and isro command while disabling the rest of Four Letter Words command:

                4lw.commands.whitelist=stat, ruok, conf, isro
              
If you really need enable all four letter word commands by default, you can use the asterisk option so you don't have to include every command one by one in the list. As an example, this will enable all four letter word commands:

                4lw.commands.whitelist=*

d2iq-mergebot · 2019-07-18T13:36:45Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T13:37:13Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

d2iq-mergebot · 2019-07-18T13:54:26Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-18T13:54:53Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

jgehrcke · 2019-07-18T14:51:53Z

@mesosphere-mergebot label Ready For Review

jgehrcke · 2019-07-18T20:49:54Z

Bumped kazoo, rebased on current master, force-pushed

d2iq-mergebot · 2019-07-18T20:50:15Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

d2iq-mergebot · 2019-07-19T10:58:18Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-19T10:58:21Z

Your pull request's branch is not based on the most recent version of master. Please rebase your changes against this repo's master branch.

d2iq-mergebot · 2019-07-19T10:58:38Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-19T10:59:06Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

d2iq-mergebot · 2019-07-23T13:28:04Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-23T13:28:06Z

Your pull request's branch is not based on the most recent version of master. Please rebase your changes against this repo's master branch.

Otherwise we get cp: cannot create regular file '/opt/mesosphere/packages/exhibitor--XX/usr/zookeeper/lib/': Not a directory

Otherwise Exhibitor cannot detect ZK health, error message: Command ruok is not executed because it is not in the whitelist.

Quote from ZK docs: The AdminServer is an embedded Jetty server that provides an HTTP interface to the four letter word commands. By default, the server is started on port 8080, and commands are issued by going to the URL "/commands/[command name]", e.g., http://localhost:8080/commands/stat. The command response is returned as JSON In DC/OS, do not expose this on all interfaces, but instead accept connections from localhost only. Also, use a port different from 8080 (this is Root Marathon's port).

Explicitly listen on IPv4 interface. Config var is documented with: New in 3.3.0: the address (ipv4, ipv6 or hostname) to listen for client connections; that is, the address that clients attempt to connect to. This is optional, by default we bind in such a way that any connection to the clientPort for any address/interface/nic on the server will be accepted.

d2iq-mergebot · 2019-07-23T13:37:56Z

@mesosphere-mergebot bump-ee

d2iq-mergebot · 2019-07-23T13:38:21Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 updated.

d2iq-mergebot · 2019-07-23T13:39:39Z

Enterprise Bump mesosphere/dcos-enterprise/pull/6194 unchanged and not updated.

d2iq-mergebot · 2019-07-31T00:00:05Z

👋 This PR has been inactive for 7 days. Please review the status checks to see what needs to be done to move it forward or change the label to Work in Progress. Thank you!

jgehrcke · 2019-08-20T11:42:20Z

(we found a problem while working on this patch: https://issues.apache.org/jira/browse/ZOOKEEPER-3466)

adamtheturtle

Review added to remove this from my review requests list in GitHub.

d2iq-mergebot added the Work In Progress label Jul 18, 2019

d2iq-mergebot added Ready For Review and removed Work In Progress labels Jul 18, 2019

jgehrcke requested review from adamtheturtle, jongiddy, timaa2k, mhrabovcin and artemharutyunyan July 18, 2019 14:59

jgehrcke added 11 commits July 23, 2019 15:37

zookeeper: bump from 3.4.14 to 3.5.5

bddd578

CHANGES.md: add ZK note

0981596

zk build: debug

672caf6

exhibitor build: fix mkdir -p

181ac78

Otherwise we get cp: cannot create regular file '/opt/mesosphere/packages/exhibitor--XX/usr/zookeeper/lib/': Not a directory

zookeeper: use binary release for 3.5.5 (incl jar files)

a51c88c

exhibitor: adjust build file for supporting zk 3.5

b8e698a

exhibitor: add 4lw.commands.whitelist=* to ZK config

0c66363

Otherwise Exhibitor cannot detect ZK health, error message: Command ruok is not executed because it is not in the whitelist.

kazoo: bump from 2.4.0 to 2.6.1

6590bce

exhibitor: set log level to TRACE (WIP)

1657613

jgehrcke force-pushed the jp/bump-to-zk-35 branch from 1ea4f68 to 1657613 Compare July 23, 2019 13:37

jgehrcke added Holding and removed Ready For Review labels Aug 2, 2019

Slach mentioned this pull request Jan 15, 2020

According 31z4/zookeeper-docker#86

Closed

adamtheturtle reviewed Feb 21, 2020

View reviewed changes

jongiddy mentioned this pull request Sep 2, 2020

Upgrade to ZooKeeper 3.5.8 #7595

Closed

DCOS_OSS-5277: Explore switching to ZooKeeper 3.5.x for DC/OS 1.14 #5878

Are you sure you want to change the base?

DCOS_OSS-5277: Explore switching to ZooKeeper 3.5.x for DC/OS 1.14 #5878

Conversation

jgehrcke commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

jgehrcke commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

adamtheturtle commented Jul 18, 2019

adamtheturtle commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

jgehrcke commented Jul 18, 2019

jgehrcke commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

jgehrcke commented Jul 18, 2019

jgehrcke commented Jul 18, 2019

d2iq-mergebot commented Jul 18, 2019

d2iq-mergebot commented Jul 19, 2019

d2iq-mergebot commented Jul 19, 2019

d2iq-mergebot commented Jul 19, 2019

d2iq-mergebot commented Jul 19, 2019

d2iq-mergebot commented Jul 23, 2019

d2iq-mergebot commented Jul 23, 2019

d2iq-mergebot commented Jul 23, 2019

d2iq-mergebot commented Jul 23, 2019

d2iq-mergebot commented Jul 23, 2019

d2iq-mergebot commented Jul 31, 2019

jgehrcke commented Aug 20, 2019

adamtheturtle left a comment

Choose a reason for hiding this comment