SpectatorD - A High Performance Metrics Daemon

SpectatorD is a high-performance telemetry daemon that listens for metrics specified by a text-based protocol and publishes updates periodically to an Atlas aggregator service. It consolidates the logic required to apply common tagging to all metrics received, maintain metric lifetimes, and route metrics to the correct backend.

Command Line Configuration Flags

spectatord --help
spectatord: A daemon that listens for metrics and reports them to Atlas.

    --age_gauge_limit (The maximum number of age gauges that may be reported by
      this process.); default: 1000;
    --common_tags (Common tags: nf.app=app,nf.cluster=cluster. Override the
      default common tags. If empty, then spectatord will use the default set.
      This flag should only be used by experts who understand the risks.);
      default: "";
    --debug (Debug spectatord. All values will be sent to a dev aggregator and
      dropped.); default: false;
    --enable_socket (Enable UNIX domain socket support. Default is true on Linux
      and false on MacOS.); default: true;
    --enable_statsd (Enable statsd support.); default: false;
    --meter_ttl (Meter TTL: expire meters after this period of inactivity.);
      default: 15m;
    --port (Port number for the UDP socket.); default: 1234;
    --socket_path (Path to the UNIX domain socket.);
      default: "/run/spectatord/spectatord.unix";
    --statsd_port (Port number for the statsd socket.); default: 8125;
    --admin_port (Port number for the admin server.); default: 1234;
    --uri (Optional override URI for the aggregator.); default: "";
    --verbose (Use verbose logging.); default: false;
    --verbose_http (Output debug info for HTTP requests.); default: false;

Try --helpfull to get a list of all flags or --help=substring shows help for
flags which include specified substring in either in the name, or description or
path.

Endpoints

By default, the daemon will listen on the following endpoints:

• Metrics Message Protocol
  • 1234/udp (~430K reqs/sec with 16MB buffers)
  • /run/spectatord/spectatord.unix Domain Socket (~1M reqs/sec with batching)
• Admin Server: 1234/tcp

The choice of which endpoint to use is determined by your performance and access requirements; the Unix domain socket offers higher performance, but requires filesystem access, which may not be tenable under some container configurations. See Performance Numbers for more details.

Usage Examples

⚠️ In container environments, the -w0 option may not work and -w1 should be used instead.

echo "c:server.numRequests,id=failed:1" | nc -u -w0 localhost 1234
echo "t:server.requestLatency:0.042" | nc -u -w0 localhost 1234
echo "d:server.responseSizes:1024" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "g:someGauge:60" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "g,300:anotherGauge:60" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "X,1543160297100:monotonic.Source:42" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "X,1543160298100:monotonic.Source:43" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "A:age.gauge:0" | nc -u -w0 localhost 1234

Message Format

The message sent to the server has the following format, where the ,options and ,tags portions are optional:

metric-type,options:name,tags:value

Multiple lines may be sent in the same packet, separated by newlines (\n):

echo -e "t:server.requestLatency:0.042\nd:server.responseSizes:1024" | nc -u -w0 localhost 1234

Metric Types

Symbol	Metric Type	Description
c	Counter	The value is the number of increments that have occurred since the last time it was recorded.
d	Distribution Summary	The value tracks the distribution of events. It is similar to a Timer, but more general, because the size does not have to be a period of time. For example, it can be used to measure the payload sizes of requests hitting a server or the number of records returned from a query.
g	Gauge	The value is a number that was sampled at a point in time. The default time-to-live (TTL) for gauges is 900 seconds (15 minutes) - they will continue reporting the last value set for this duration of time. Optionally, the TTL may be specified in seconds, with a minimum TTL of 5 seconds. For example, g,120:gauge:42.0 spcifies a gauge with a 120 second (2 minute) TTL.
m	Max Gauge	The value is a number that was sampled at a point in time, but it is reported as a maximum gauge value to the backend.
t	Timer	The value is the number of seconds that have elapsed for an event.
A	Age Gauge	The value is the time in seconds since the epoch at which an event has successfully occurred, or 0 to use the current time in epoch seconds. After an Age Gauge has been set, it will continue reporting the number of seconds since the last time recorded, for as long as the spectatord process runs. The purpose of this metric type is to enable users to more easily implement the Time Since Last Success alerting pattern. To set a specific time as the last success: A:time.sinceLastSuccess:1611081000. To set now() as the last success: A:time.sinceLastSuccess:0. By default, a maximum of 1000 Age Gauges are allowed per spectatord process, because there is no mechanism for cleaning them up. This value may be tuned with the --age_gauge_limit flag on the spectatord binary.
C	Monotonic Counter	The value is a monotonically increasing number. A minimum of two samples must be received in order for spectatord to calculate a delta value and report it to the backend. A variety of networking metrics may be reported monotically and this metric type provides a convenient means of recording these values, at the expense of a slower time-to-first metric.
D	Percentile Distribution Summary	The value tracks the distribution of events, with percentile estimates. It is similar to a Percentile Timer, but more general, because the size does not have to be a period of time. For example, it can be used to measure the payload sizes of requests hitting a server or the number of records returned from a query. In order to maintain the data distribution, they have a higher storage cost, with a worst-case of up to 300X that of a standard Distribution Summary. Be diligent about any additional dimensions added to Percentile Distribution Summaries and ensure that they have a small bounded cardinality.
T	Percentile Timer	The value is the number of seconds that have elapsed for an event, with percentile estimates. This metric type will track the data distribution by maintaining a set of Counters. The distribution can then be used on the server side to estimate percentiles, while still allowing for arbitrary slicing and dicing based on dimensions. In order to maintain the data distribution, they have a higher storage cost, with a worst-case of up to 300X that of a standard Timer. Be diligent about any additional dimensions added to Percentile Timers and ensure that they have a small bounded cardinality.
X	Monotonic Counter with Millisecond Timestamps	The value is a monotonically increasing number, sampled at a specified number of milliseconds since the epoch. A minimum of two samples must be received in order for spectatord to calculate a delta value and report it to the backend. This is an experimental metric type that can be used to track monotonic sources that were sampled in the recent past, with the value normalized over the reported time period. The timestamp in milliseconds since the epoch when the value was sampled must be included as a metric option: X,1543160297100:monotonic.Source:42

Metric Name and Tags

The metric name and tags must follow Atlas restrictions, which are described in the sections below.

Tags are optional. They may be specified as comma-separated key=value pairs after the metric name. For example:

fooIsTheName,some.tag=val1,some.otherTag=val2

See Atlas Naming Conventions for recommendations on naming metrics.

Length Restrictions

Limit	Min	Max
Length of name	1	255
Tag key length	2	60
Tag value length	1	120

Allowed Characters

The metric name, tag keys and values may only use characters in the following set: -._A-Za-z0-9.

All others characters will be converted to an underscore (_) by the client.

To avoid issues with parsing metrics, avoid using the SpectatorD protocol delimiter characters (,=:) rather than relying on the client to rewrite them to _.

Metric Value

A double value. The meaning of the value depends on the metric type.

Metrics

See METRICS for a list of metrics published by this service.

Admin Server

An administrative server is provided with SpectatorD, so that debugging information and few data management tasks may be completed. By default, this server listens on port 1234/TCP, but this can be modified with the --admin_port flag. The endpoints which change data may only be accessed from localhost.

• GET /
  • Returns a service description and list of available endpoints.
• GET /config
  • Returns the current SpectatorD configuration, including the current set of common tags.
• GET /config/common_tags
  • Returns a description of how to use this endpoint to modify common tags.
• POST /config/common_tags
  • Create, modify or delete common tags from the allowed set of Mantis common tags. No other common tags may be modified. Create or update a tag by setting it to a string. Delete a tag by setting the value to an empty string.
  • Allowed tags:
    • mantisJobId
    • mantisJobName
    • mantisUser
    • mantisWorkerIndex
    • mantisWorkerNumber
    • mantisWorkerStageNumber
  • Example:

    curl -X POST \
    -d '{"mantisJobId": "foo", "mantisJobName": "bar", "mantisUser": ""}' \
    -w " %{http_code}\n" \
    http://localhost:1234/config/common_tags
    

• GET /metrics
  • Return an object containing lists of all metrics currently known to the Registry, grouped by type.
• DELETE /metrics/A
  • Delete all AgeGauge metrics from the Registry.
• DELETE /metrics/A/{id}
  • Delete one AgeGauge metric from the Registry, identified by the id.
  • Example:

    curl -X DELETE \
    -w " %{http_code}\n" \
    http://localhost:1234/metrics/A/fooIsTheName,some.tag=val1,some.otherTag=val2
    

• DELETE /metrics/g
  • Delete all Gauge metrics from the Registry.
• DELETE /metrics/g/{id}
  • Delete one Gauge metric from the Registry, identified by the id.
  • Example:

    curl -X DELETE \
    -w " %{http_code}\n" \
    http://localhost:1234/metrics/g/fooIsTheName,some.tag=val1,some.otherTag=val2
    

Performance Numbers

A key goal of this project is to deliver high performance. This means that we need to use few resources for the common use case, where the number of metric updates is relatively small (< 10k reqs/sec), and it also needs to be able to handle hundreds of thousands of updates per second when required.

Using Unix domain sockets, we can handle close to 1M metric updates per second, assuming the client batches the updates and sends a few at a time. Sending every single metric update requires a lot of context switching, but is something that works well for the majority of our use cases. This simplicity means the user does not have to maintain any local state.

Transport          Batch Size    First 10M          Second 10M
Unix Dgram         1             22.98s (435k rps)  20.58s (486k rps)
Unix Dgram         8             11.46s (873k rps)   9.89s (1011k rps)
Unix Dgram         32            10.38s (963k rps)   8.49s (1178k rps)

The UDP transport is particularly sensitive the max receive buffer size (16MB on our systems).

Our tests indicate that sending 430K rps to the UDP port did not drop packets, but if there is a need for higher throughput, then tweaking /proc/sys/net/unix/max_dgram_qlen is recommended.

Performance Testing

• Start spectatord in debug mode (--debug) to send metrics to a dev stack of the Atlas aggregator, which will perform validation and return the correct HTTP status codes for payloads, then drop the metrics on the floor. Alternatively, you can also configure it to send metrics to /dev/null.
• Use the metrics_gen binary to generate and send a stream of metrics to a running spectatord binary.
• Use the perf-record and perf-report Linux utilities to measure the performance of the running binary.
• The udp_numbers.pl script is used to automate running metrics_gen with different kernel settings for UDP sockets.

Local Development

# setup python venv and activate, to gain access to conan cli
./setup-venv.sh
source venv/bin/activate

# link clion default build directory to our build directory
ln -s cmake-build cmake-build-debug

./build.sh  # [clean|clean --force|skiptest]

• CLion > Preferences > Plugins > Marketplace > Conan > Install
• CLion > Preferences > Build, Execution, Deploy > Conan > Conan Executable: $PROJECT_HOME/venv/bin/conan