RemoveToxic: HTTP 503: Unexpected response code, expected 204

[sergey-kurenkov]

Sometimes (not always) I receive this error.

This is an example from my log:

...
gengw_mdt Added upstream reset_peer toxic 'reset_peer_upstream' on proxy 'gengw_mdt'
gengw_mdt Added upstream timeout toxic 'timeout_upstream' on proxy 'gengw_mdt'
gengw_mdt Removed toxic 'timeout_upstream' on proxy 'gengw_mdt'
gengw_mdt Removed toxic 'reset_peer_upstream' on proxy 'gengw_mdt'
gengw_mdt: going to sleep for 394 seconds
gengw_mdt: adding reset_peer toxicity
gengw_mdt Added downstream reset_peer toxic 'reset_peer_downstream' on proxy 'gengw_mdt'
gengw_mdt Removed toxic 'reset_peer_downstream' on proxy 'gengw_mdt'
gengw_mdt: going to sleep for 459 seconds
gengw_mdt: adding bandwidth_50kb toxicity for 246 seconds
gengw_mdt Added downstream bandwidth toxic 'bandwidth_downstream' on proxy 'gengw_mdt'
Failed to remove toxic: failed to remove toxic 'bandwidth_downstream' from proxy 'gengw_mdt': RemoveToxic: HTTP 503: Unexpected response code, expected 204
gengw_mdt
gengw_mdt: going to sleep for 512 seconds
gengw_mdt: adding reset_peer toxicity
Failed to add toxic: failed to retrieve proxy with name `gengw_mdt`: Proxy: HTTP 503: Unexpected response code, expected 200
gengw_mdt
Failed to remove toxic: failed to retrieve proxy with name `gengw_mdt`: Proxy: HTTP 503: Unexpected response code, expected 200
gengw_mdt
gengw_mdt: going to sleep for 338 seconds
gengw_mdt: adding bandwidth_50kb toxicity for 170 seconds
Failed to add toxic: failed to retrieve proxy with name `gengw_mdt`: Proxy: HTTP 503: Unexpected response code, expected 200
gengw_mdt
Failed to remove toxic: failed to retrieve proxy with name `gengw_mdt`: Proxy: HTTP 503: Unexpected response code, expected 200
gengw_mdt

[sergey-kurenkov]

/tp # ./toxiproxy-cli --version
toxiproxy-cli version 2.5.0
/tp # ./toxiproxy --version
toxiproxy-server version 2.5.0
/tp # 

[sergey-kurenkov]

This is a config file:

[
  {
    "name": "gengw_svc",
    "listen": "0.0.0.0:19102",
    "upstream" : "localhost:9102",
    "enabled": true
  },
  {
    "name": "gengw_mdt",
    "listen": "0.0.0.0:19103",
    "upstream" : "localhost:9103",
    "enabled": true
  }
]

[miry]

@sergey-kurenkov It is not clear how toxiproxy-server is deployed from your example. I see those only client logs. Would like to provide a example how I could reproduce the problem on local machine?

[miry]

hm, actually it could be related to https://github.com/Shopify/toxiproxy/pull/422/files
It means there is a resource locked, that prevent to remove resource.

Context:

In 2.4 it was a report when using of Java client the toxiproxy was stuck to remove specific proxy. It was some sort race condition, when mutating the proxy and traffic is going on.
It was introduced few PRs: return api response as fast as possible 503 and force drop connection after some time.

[miry]

cc: @dianadevasia