Error: Server at xyz not available\n at Client.memcachedCommand [as command] in nodejs with Memcached

[cmreddyv]

getting the error:

Getting data from memCache failed severity:error f.err:{"stack":"Error: Server at xyz not available\n at Client.memcachedCommand [as command] (/service/node_modules/memcached/lib/memcached.js:306:70)\n at Client.get (/service/node_modules/memcached/lib/memcached.js:830:10)\n at /service/repository/data/Repo.js:32:35\n at new Promise ()\n at Repo.get (/service/repository/data/Repo.js:13:16)\n

Client which is connecting is npm memcached
Twemproxy version: 0.5.0
Environemnt: Running twemproxy inside kubernetes

When checked for twemproxy stats no server ejections all the memcached servers are operating normally.

Any lead on how to solve the above error would be really helpful

[TysonAndre]

1. Can you post the config file used for twemproxy with as much detail as possible?
2. Did you disable tls for the elasticache cluster? - See Does twemproxy support connection to tls redis server ? #583 - twemproxy currently does not support tls (for memcached and redis)
3. Are you able to connect to those memcache hosts directly from the kubernetes cluster?
4. Can you post logs, preferably at loglevel 6(informational) or a numerically larger value (nutcracker -v 6 -o /var/log/nutcracker.log)

[cmreddyv]

Twemproxy conf file

memcached:
listen: 0.0.0.0:22121
hash: fnv1a_64
distribution: ketama
preconnect: true
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 2
timeout: 1000
tcpkeepalive: true
backlog: 1024
servers:
- 0.xyz-service.svc.cluster.local:11211:1 0
- 1.xyz-service.svc.cluster.local:11211:1 1
- 2.xyz-service.svc.cluster.local:11211:1 2
- 3.xyz-service.svc.cluster.local:11211:1 3
- 4.xyz-service.svc.cluster.local:11211:1 4
- 5.xyz-service.svc.cluster.local:11211:1 5

We are not using elasticcache cluster... it is like having a onprem memcached on kubernetes and twemproxy in kubernetes.

Yes i am able to connect to memcached hosts.

Strange part is after some time, not sure about the time, the error is going away... looks like the server reconnect feature.

When i checked the npm module code

https://github.com/3rd-Eden/memcached/blob/master/lib/memcached.js

Line number: 311,312,313 this the causing the error.

Is there anything to do with the server retries or our config is correct and we have set the following in the client

poolsize: 70
retries: 3,
retry: 1000,
reconnect: 60000

Please guide us if we are using the correct twem proxy config and client config

[TysonAndre]

Do you start twemproxy and your application at the same time? Are you sure twemproxy has finished starting up when you connect to it (should finish in a few seconds)

Also, what do the logs from twemproxy say (can be enabled with nutcracker -v 6 -o /path/to/logfile.log)