SPIKE: Waltz admin UI

[whynick1]

Waltz needs an admin UI. UI should initially be read-only, and should allow us to visualize the state of the cluster. Storage nodes, partitions, latency, offline/online reads, etc.

We should do this work once the admin CLI tools are completed.

This ticket is to spec out what an M0 Waltz admin UI would look like.

[whynick1]

Talked with Matthew Clower about the possibility of use Grafana as Waltz admin UI.

Using Grafana provides a decent decoupling between UI and service. Also, significantly simply the job of displaying Waltz component status (server node, storage node, partition), especially considering we already integrated Metrics JMX reporter with Waltz.

The original purpose of admin UI is to provide an easy for monitoring. So I checked existing Grafana panel that fit our requirement, and found this Status Group By Plugin. Base on which, we can build panel with flowchart be even better.

However, there are two concerns:

1. Major: Grafana Status Group By Panel doesn't support dynamic grouping

Metrics in a group is static. This means, Grafana cannot dynamic update panel layout reflecting metrics change (etc. dynamically adding new panel for new storage node; auto shuffle metrics group according to Waltz partition assignment change)

2. Minor: Grafana doesn't support complex query

It means if we want to determine if a partition reach quorum, we have to put that assert logic either in Waltz or in prometheus, which is fine.

As a summary, there is a lot of benefit of using Grafana, however, if we want to have an dynamic admin UI with clear relationship among server, storage and partition, we still need to build our own UI.

[whynick1]

Talked to @criccomini , we plan to do some spike regarding creating our own Grafana panel plugin to meet our need. (to leverage our existing tool chain if possible)

Here are some useful links:
https://github.com/grafana/grafana-plugin-repository
http://docs.grafana.org/plugins/developing/development/

[whynick1]

Found an example drawing a collapsible tree (https://observablehq.com/@d3/collapsible-tree).

Kind of similar to Orchestrator (http://tst-immutable-orchestrator.devops.wepay-inc.com:3000/web/cluster/tst-immutable-useast1a-mysql01:53306#)