Consul sometimes during resilience tests goes to strange state

[slonka]

Author: @lukidzi

When we run resilience tests in specific order we might observe that consul is in strange state
during test should be resilient to transient unavailability of one DC:

2019/05/30 12:16:49 [DEBUG] memberlist: Stream connection from=192.168.144.10:33118
    2019/05/30 12:16:49 [WARN] consul.rpc: RPC request for DC "dc2", no path found
    2019/05/30 12:16:49 [ERR] http: Request GET /v1/health/service/envoy-control?passing&dc=dc2, error: No path to datacenter from=192.168.144.1:34266
    2019/05/30 12:16:49 [DEBUG] http: Request GET /v1/health/service/envoy-control?passing&dc=dc2 (2.285225ms) from=192.168.144.1:34266
    2019/05/30 12:16:49 [WARN] consul.rpc: RPC request for DC "dc2", no path found

This state keeps for ~2 min. Only restart of container help (consul restart).

I tried to kill all connections during connections cut off but it didn't help. Also restart of interfaces and change arp gc config didn't help.

We should write test to reproduce this state and report the problem to Hashicorp.