Not able to deploy mr-redis Cluster on mesos.

[deepakkashyap]

OUTPUT:

curl -X "POST" mrredis.mesos:5656/v1/CREATE/redisCluster/100/1/2
2016/11/29 17:21:35 Instance Name=redisCluster, Capacity=100, masters=1, slaves=2 ConfigJson={1}
ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/
ZOO K=/MrRedis/Instances/redisCluster/Type V=MS
ZOO K=/MrRedis/Instances/redisCluster/Masters V=0
ZOO K=/MrRedis/Instances/redisCluster/Slaves V=0
ZOO K=/MrRedis/Instances/redisCluster/Capacity V=100
ZOO K=/MrRedis/Instances/redisCluster/ExpMasters V=1
ZOO K=/MrRedis/Instances/redisCluster/ExpSlaves V=2
ZOO K=/MrRedis/Instances/redisCluster/Status V=CREATING
ZOO K=/MrRedis/Instances/redisCluster/Mname V=
ZOO K=/MrRedis/Instances/redisCluster/DValue V=1
ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/Snames/
ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/Procs/
2016/11/29 17:21:35 [router.go:854] [D] | POST | /v1/CREATE/redisCluster/100/1/2 | 56.157575ms | match | /v1/CREATE/:INSTANCENAME/:CAPACITY/:MASTERS/:SLAVES |
2016/11/29 17:21:35 Received offer {true 0xc4200de360 1}
Request Accepted, Instance will be created.2016/11/29 17:21:35 Created 1 master offers for Instance redisCluster
root@mesos-master1:/usr/local/go/src/github.com/mesos/mr-redis/sched# 2016/11/29 17:21:35 Received Offer with CPU=7.5 MEM=13982 OfferID=215731b7-c3e2-4ad9-8333-c3c7231af861-O5849452
2016/11/29 17:21:35 Launched 1 tasks from this offer
2016/11/29 17:21:35 Received Offer with CPU=8 MEM=14014 OfferID=215731b7-c3e2-4ad9-8333-c3c7231af861-O5849453
2016/11/29 17:21:35 Launched 0 tasks from this offer
2016/11/29 17:21:35 MrRedis Receives offer
2016/11/29 17:21:35 MrRedis Task Update received
2016/11/29 17:21:35 Status={redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 TASK_ERROR 215731b7-c3e2-4ad9-8333-c3c7231af861-S3 []}
2016/11/29 17:21:35 Received a Task update from the channel &{redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 TASK_ERROR 215731b7-c3e2-4ad9-8333-c3c7231af861-S3 []}
2016/11/29 17:21:35 Invalid Key /MrRedis/Instances/redisCluster/Procs/92b8edb7-96ee-4f1e-5f28-58a1a50b11a1, Cannot load
2016/11/29 17:21:35 Invalid Key /MrRedis/Instances/redisCluster/Procs/, Cannot load
2016/11/29 17:21:35 Task redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 is Error

curl hostname:port/v1/STATUS/redisCluster
{"Name":"redisCluster","Type":"MS","Status":"CREATING","Capacity":100,"Master":null,"Slaves":null}

Not able to debug whats wrong I'm doing.Can you please help?

[dhilipkumars]

Thanks for reporting, it is always useful with the Logs. Clearly something is wrong with the ZooKeeper drivers.
Looks like the Executor tries to write to Zookeeper and that write is not successful, do you have the executor logs? I can try and replicate this in my environment.

Is the zookeeper reachable from the slaves?

[deepakkashyap]

Yes, zookeeper is reachable
Logs:
Starting MrRedis Executor
ZOO CREATE SECTION K=/MrRedis
ZOO CREATE SECTION K=/MrRedis/Instances
ZOO CREATE SECTION K=/MrRedis/Config
InterfaceAddress = x.x.x.x Executor process has started and running.
Registered Executor on slave
Launching task trial::8ff66757-ea0d-4ca7-4a63-8dd870008369 with command
The Redmon object = {0xc8201c2000 0 x.x.x.x 6380 false 0xc8200942a0 0xc82000aa50 redis:3.0-alpine 0xc820170640}
Error creating a container error during connect: Post https://%2Fvar%2Frun%2Fdocker.sock/v1.22/containers/create?name=8ff66757-ea0d-4ca7-4a63-8dd870008369: http: server gave HTTP response to HTTPS client
Total tasks launched 1
Executor Finished, Delete all the containers
executor terminated

**I1208 18:23:16.718339 114025 fetcher.cpp:424] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/slaves/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"http://x.x.x.x:8090/MrRedisExecutor"}}],"sandbox_directory":"/data/mesos/slaves/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22/frameworks/83371dc2-4360-42c6-bb7c-35006d4de0fe-1640/executors/default/runs/3ce8e1c8-09f4-4bbd-bc87-3aa33fbd11cf","user":"root"}
I1208 18:23:16.719645 114025 fetcher.cpp:379] Fetching URI 'http://x.x.x.x:8090/MrRedisExecutor'
I1208 18:23:16.719657 114025 fetcher.cpp:250] Fetching directly into the sandbox directory
I1208 18:23:16.719669 114025 fetcher.cpp:187] Fetching URI 'http://x.x.x.x:8090/MrRedisExecutor'
I1208 18:23:16.719676 114025 fetcher.cpp:134] Downloading resource from 'http://x.x.x.x:8090/MrRedisExecutor' to '/data/mesos/slaves/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22/frameworks/83371dc2-4360-42c6-bb7c-35006d4de0fe-1640/executors/default/runs/3ce8e1c8-09f4-4bbd-bc87-3aa33fbd11cf/MrRedisExecutor'
I1208 18:23:16.886005 114025 fetcher.cpp:456] Fetched 'http://x.x.x.x:8090/MrRedisExecutor' to '/data/mesos/slaves/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22/frameworks/83371dc2-4360-42c6-bb7c-35006d4de0fe-1640/executors/default/runs/3ce8e1c8-09f4-4bbd-bc87-3aa33fbd11cf/MrRedisExecutor'
2016/12/08 18:23:16 Connected to x.x.x.x:2181
2016/12/08 18:23:16 Authenticated: id=96998585951780933, timeout=40000
I1208 18:23:16.945909 114030 executor.go:133] Init mesos executor driver
I1208 18:23:16.945921 114030 executor.go:134] Protocol Version: 0.24.0
I1208 18:23:16.945996 114030 executor.go:524] Starting the executor driver
I1208 18:23:16.946038 114030 http_transporter.go:383] listening on x.x.x.x port 34061
I1208 18:23:16.946104 114030 executor.go:559] Mesos executor is started with PID= executor(1)@x.x.x.x:34061
I1208 18:23:16.946113 114030 executor.go:626] Waiting for the executor driver to stop
I1208 18:23:16.948471 114030 executor.go:305] Executor driver registered
I1208 18:23:16.948479 114030 executor.go:318] Registered on slave &SlaveID{Value:*83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],}
I1208 18:23:16.950577 114030 executor.go:402] Executor driver runTask
I1208 18:23:16.950593 114030 executor.go:416] Executor asked to run task '&TaskID{Value:*trial::8ff66757-ea0d-4ca7-4a63-8dd870008369,XXX_unrecognized:[],}'
I1208 18:23:23.577681 114030 executor.go:687] Executor sending status update &StatusUpdate{FrameworkId:&FrameworkID{Value:*83371dc2-4360-42c6-bb7c-35006d4de0fe-1640,XXX_unrecognized:[],},ExecutorId:&ExecutorID{Value:*default,XXX_unrecognized:[],},SlaveId:&SlaveID{Value:*83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],},Status:&TaskStatus{TaskId:&TaskID{Value:*trial::8ff66757-ea0d-4ca7-4a63-8dd870008369,XXX_unrecognized:[],},State:*TASK_ERROR,Data:nil,Message:nil,SlaveId:&SlaveID{Value:*83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],},Timestamp:*1.481201603e+09,ExecutorId:nil,Healthy:nil,Source:nil,Reason:nil,Uuid:nil,Labels:nil,ContainerStatus:nil,XXX_unrecognized:[],},Timestamp:1.481201603e+09,Uuid:[77 185 135 129 189 69 17 230 154 65 6 241 36 82 183 249],LatestState:nil,XXX_unrecognized:[],}
I1208 18:23:23.580006 114030 executor.go:445] Executor statusUpdateAcknowledgement
I1208 18:23:23.580015 114030 executor.go:448] Receiving status update acknowledgement &StatusUpdateAcknowledgementMessage{SlaveId:&SlaveID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],},FrameworkId:&FrameworkID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-1640,XXX_unrecognized:[],},TaskId:&TaskID{Value:trial::8ff66757-ea0d-4ca7-4a63-8dd870008369,XXX_unrecognized:[],},Uuid:[77 185 135 129 189 69 17 230 154 65 6 241 36 82 183 249],XXX_unrecognized:[],}
I1208 18:43:24.014788 114030 executor.go:489] Executor driver received shutdown
I1208 18:43:24.014807 114030 executor.go:496] Executor driver is asked to shutdown
I1208 18:43:24.014816 114030 executor.go:572] Stopping the executor driver
I1208 18:43:24.014822 114030 messenger.go:295] stopping messenger executor(1)@x.x.x.x:34061..
I1208 18:43:24.014838 114030 http_transporter.go:447] stopping HTTP transport

[dhilipkumars]

@deepakkashyap This is interesting. Mr-redis comes with custom executor, it appears that it is unable to connect to the docker daemon at the mesos-agent.

The customer executor will automatically perform a docker pull by communicating with the docker daemon, and then perform a docker run.

[dhilipkumars]

in the config file , could you supply the actual "username" which has permission to access the docker unix:///var/run/docker.sock socket file? If you have installed docker using a sudo command then its safe to supply 'root' for the UserName attribute in the config file.

$cat config.json
{
   "UserName": "root",
   "FrameworkName": "MrRedis",
   "Master": "127.0.0.1:5050",
   "ExecutorPath": "./MrRedisExecutor",
   "RedisImage": "redis:3.0-alpine",
   "DBType": "zookeeper",, 
   "DBEndPoint": "127.0.0.1:2181"", 
   "DBType": "etcd", 
   "DBEndPoint": "127.0.0.1:2379", 
   "LogFile": "stderr",
   "ArtifactIP": "127.0.0.1",
   "ArtifactPort": "5454",
   "HTTPPort": "5656"
}

Ofcourse I'm assume that docker daemon is pre-installed on all your mesos Agents. :-)

[deepakkashyap]

I am running framework as root user only.
Config file:
{
"UserName": "root",
"Master": "127.0.0.1:5050",
"FrameworkName": "MrRedis",
"ArtifactIP": "127.0.0.1",
"ExecutorPath": "/usr/local/go/src/github.com/mesos/mr-redis/exec/MrRedisExecutor",
"RedisImage": "redis:3.0-alpine",
"ArtifactPort": "8089",
"HTTPPort": "5656",
"DBType": "zookeeper",
"DBEndPoint": "127.0.0.1:2181"
}

[dhilipkumars]

@deepakkashyap what is the docker version running on the 'Agent'? Do you have any redis containers already running on that slave?

$docker ps 

[deepakkashyap]

Docker version is 1.11.2
and Redis container is not running on any of the slave.