Monitor SWF rate limits

[jbbarth]

Recently at Botify we hit situations where we would overflow our SWF rate limits, especially in RecordActivityTaskHeartbeat and PollForActivityTask operations. The main problem for me is that we hit a certain scale where we're subject to those limits but we don't know at all what we're consuming. As far as I can tell, Amazon doesn't give us any hint into our current rate limit consumption, nor it has Cloudwatch metrics associated with it.

I opened a support request to increase the quotas for our own account, which will drive the priority of this issue (sorry that sounds shitty but let's be pragmatic). I'll report here.

Anyway, having those values monitored via a client-side integration in simpleflow would be nice. The visualisation part could happen in webflow later.

My first thoughts on this:

• we still don't want a server in simpleflow, so the simplest design is to hit a datastore (a ubiquitous one like Redis would be nice)
• this post has some ideas about storing timeseries data in Redis in a "RRD" fashion
• the first goal is to have counters so we could simply use HINCRBY on hash like <account_id>:<region>:<service>:<endpoint> with keys like <timestamp_in_seconds> ; example: HINCR 64342123456:us-east-1:swf:RecordActivityTaskHeartbeat:by_second 2017-09-05T07:28:21Z and have details stored elsewhere (possibly using transactions via MULTI/EXEC

To be continued...