WiP - API Throttling

[akutz]

This patch fixes #959 and introduces API throttling in an alpha state for the purposes of code/workflow review. This feature depends on etcd v3 for persistent, external storage -- responsible for tracking active connections and associating them with sustained and burst TTL leases.

TODO

• Proof-of-concept
• External configuration
  • etcd connection info
  • Burst throttling data
  • Sustained throttling data
• Refactor into service used by handler
• Refactor workflow to inject/pull etcd client into/from Context

etcd

This section reviews how to install, configure, and use etcd.

Install etcd

To install etcd and etcdctl type:

$ go get -u github.com/coreos/etcd/cmd/etcd github.com/coreos/etcd/cmd/etcdctl

Configure etcd

Please note that in a terminal in which etcd or etcdctl may be executed, the following environment variable should be set:

$ export ETCDCTL_API=3

The above environment variable instructs both the etcd server and client to use v3 of its API.

Start etcd server

Once the server binary is built, start it by executing:

$ etcd

Use etcd client

Finally, the etcdctl program can be used to control etcd from the command line. Open a new terminal and type:

$ etcdctl put hello world
OK

$ etcdctl get hello
hello
world

$ etcdctl del hello
1

Basic workflow

The basic workflow for how throttling is implemented in REX-Ray is as follows:

1. If ETCD_ENDPOINTS defined then proceed.
2. Get etcd connection
3. Check sustained & burst connection count for throttling domain; if over return error
4. Get mutex for throttling domain
5. Lock mutex
6. Get sustained lease ID and burst lease ID for throttling domain
7. If one or both of the leases do not exist then create them
8. Record the lease IDs back to etcd, associating the keys with the appropriate lease. This way when the lease expires, so does its key, and that way RR knows to create a new lease
9. Check sustained & burst connection count for throttling domain AGAIN; if over return error
10. Record current request in both the sustained and burst throttling domain, associating the keys with the respected leases
11. Allow the request
12. Unlock mutex

Without ever trying to obtain a lock for the the throttling domain's mutex the code first checks to see if the connection count is already too high. If so, none of the heavy work is required. If it's passable then the mutex is retrieved and a lock attempt is made. This way it fails fast and adds a slight penalty only to proceed

Throttling Domains

Each libStorage service is associated with a throttling domain. By default the name of that domain is the name of the libStorage service. However, it's possible for libStorage services to define a custom throttling domain. For example:

libstorage:
  server:
    services:
      vfs1:
        driver: vfs
        throttling:
          domain: vfs
      vfs2:
        driver: vfs
        throttling:
          domain: vfs

In the above configuration the services vfs1 and vfs2 both belong to the vfs throttling domain.

Testing this PR

Once etcd is installed and running, simply build REX-Ray using this PR and then execute then export the following environment variable:

export ETCD_ENDPOINTS=127.0.0.1:2379

Please note that the above value can be represented in a REX-Ray configuration file:

etcd:
  endpoints:
    - 127.0.0.1:2379

Now all libStorage requests will be subject to throttling. Currently the burst and sustained values are hardcoded as such:

• sustained is 1000 allowed requests in a 24 hour period per throttling domain
• burst is 60 allowed requests per minute per throttling domain

Printing throttling stats

To list the active throttling information about REX-Ray, use etcdctl:

$ etcdctl get /rexray/throttling --prefix
/rexray/throttling/vfs/burst/cnxns/2d341609-56d1-43c5-58ff-4ee0f9888adc/1506120755

/rexray/throttling/vfs/burst/cnxns/38208d39-57bf-4ba4-7a45-3efc775fb1ca/1506120752

/rexray/throttling/vfs/burst/cnxns/3fb61bb1-c69d-4a25-5c7a-21d96a5440e6/1506120775

/rexray/throttling/vfs/burst/cnxns/70307eb3-b7be-4464-682f-f1502ce1bc9b/1506120778

/rexray/throttling/vfs/burst/cnxns/af55efca-cc65-4f46-7bb0-f5f2b011664d/1506120767

/rexray/throttling/vfs/burst/cnxns/dbf61251-3ea0-47d4-4a9d-56bbdcfd306a/1506120780

/rexray/throttling/vfs/burst/lease
7587825039093911051
/rexray/throttling/vfs/sustained/cnxns/2d341609-56d1-43c5-58ff-4ee0f9888adc/1506120755

/rexray/throttling/vfs/sustained/cnxns/38208d39-57bf-4ba4-7a45-3efc775fb1ca/1506120752

/rexray/throttling/vfs/sustained/cnxns/3fb61bb1-c69d-4a25-5c7a-21d96a5440e6/1506120775

/rexray/throttling/vfs/sustained/cnxns/70307eb3-b7be-4464-682f-f1502ce1bc9b/1506120778

/rexray/throttling/vfs/sustained/cnxns/af55efca-cc65-4f46-7bb0-f5f2b011664d/1506120767

/rexray/throttling/vfs/sustained/cnxns/dbf61251-3ea0-47d4-4a9d-56bbdcfd306a/1506120780

/rexray/throttling/vfs/sustained/lease
7587825039093911055

The key structure is as follows:

• /rexray/throttling is the prefix
• Next is the throttling domain. This is the service name by default, but any service can define rexray.throttling.domain to specify a custom throttling domain.
• Next is the throttle type. ex - sustained might be 1000/24h whereas burst is 60/1m.
• Next is the cnxns "sub-dir". It's a "sub-dir" because /rexray/throttling/vfs/sustained/lease also exists and is the ID of the lease associated with this throttle type.
• Under cnxns are the transaction IDs associated with requests.
• Transaction IDs can also have child elements -- the epoch of the request. This is the case where one transaction from the client might make multiple requests to the controller.

[codecov-io]

Codecov Report

Merging #1040 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1040   +/-   ##
=======================================
  Coverage   34.16%   34.16%           
=======================================
  Files          36       36           
  Lines        2901     2901           
=======================================
  Hits          991      991           
  Misses       1807     1807           
  Partials      103      103

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a39fd7...403c71e. Read the comment docs.