Runtime Controls

[jnunemaker]

Features that can be dynamically enabled and disabled at runtime are very powerful, but they aren't the only kinds of things that a developer/operator might want to flip at runtime. At GitHub and nearly every application I've worked on, the app stores controls in a variety of ways (env var, memcached, redis, mysql, etc.). I've noticed that having a standard way of controlling applications at runtime would be valuable.

What are controls?

Features are all about is something enabled for an actor or not. They are more on/off. Controls, on the other hand, are for things you want to control that are not as simple as on/off. Perhaps an integer, string or even a collection of things that when used in the application, change the behavior of the app.

Concrete Example

One example would be poll interval. Backends for mobile apps (and even the github API) often report back to the mobile app a poll interval header or similar to inform the mobile app when to contact the backend again. This allows shedding load during problems by increasing the poll interval and reducing traffic from the mobile app. Poll interval is a great example of something that is not on or off, but perhaps configured to a number, which represents the number of seconds for mobile apps to wait between contacting the service.

# perhaps something like this...
flipper = Flipper.new(...)
flipper.set_control(:poll_interval, 5)
flipper.control(:poll_interval) # => 5

# or even
flipper.controls[:poll_interval] # => 5

Why flipper instead of some new gem?

I considered spinning up a new gem, but then I realized all that flipper had to offer that I would have to duplicate and decided it just would not be worth it to start over. Flipper has a great base setup for controlling things at runtime:

• instrumentation - flipper's automatic instrumentation can publish performance details to statsd/graphite
• memoization - flipper and its memoizing middleware can handle ensuring that control data is only load this once per request or job
• performance - controls can be stored wherever it makes sense (on disk, consul, mysql, memcached, redis, whatever) thanks to the adapter pattern flipper uses; this means you can eek out every ounce of performance you need by using whatever backend you desire
• auditing - flipper's instrumentation makes it easy to add audit log entries for all changes (e.g. poll interval control value changed from 5 seconds to 10 seconds at this point in time)
• UI - flipper has a UI that controls could be added to which increases the visibility of the controls in a system for developers and operators

Because of the aforementioned flipper functionality, I think it makes sense to add the idea of "controls" to flipper itself. Additionally, flipper is about controlling things at runtime, not just features. I just happened to start with features.

Status

I started a branch to add controls. While working on the branch, I realized that it is inconvenient to need to modify adapters for every new bit of functionality I want to add. What might make more sense is a more generic way of storing data that can be used for features, controls and anything else we come up with that makes sense inside of flipper.

For this reason, I started venturing down the path of making adapters key/value based. This makes it dramatically easier to create adapters and increases flexibility of the adapters, which means new features can be added to flipper and persisted without needing to change adapters each time.

I'd like to hold off on shipping controls until the KV project is done. I hope to write more up about the KV project soon, but I did kick off a project board with a few internal (to me) notes for those that are curious and I'd love to get feedback on the control idea if anyone has any.