HdrHistogram_vala

Gil Tene's High Dynamic Range Histogram vala implementation

Who's better than Gil Tene himself to describe what is HdrHistogram:

HdrHistogram supports the recording and analyzing of sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording, and provides control over value quantization behavior across the value range and the subsequent value resolution at any given level.

API

Create an Histogram

When creating a histogram instance, you must provide the minimum and maximum trackable values and the wanted number of precision digits. If you need to create a histogram to record latency between one millisecond and one minute [1ms..1min] with a precision of 0.01, use the following:

var histogram = new Hdr.Histogram(1, 60000, 2); //From 1ms to 60000ms with a precision of 2 digits, 0.01

Histogram can stores counters in many bucket sizes, which impacts histogram memory footprint. This library supports unsigned 8-bit, unsigned 16-bit, unsigned 32-bit and signed 64-bit histogram counters.

var histogram = new Hdr.Int8Histogram(1, 60000, 2);
var histogram = new Hdr.Int16Histogram(1, 60000, 2);
var histogram = new Hdr.Int32Histogram(1, 60000, 2);

Record values

histogram.record_value(42);

Statistics

Percentils

histogram.record_value(45);
histogram.record_value(115);
histogram.record_value(198);
histogram.record_value(215);
histogram.record_value(320);
histogram.record_value(578);

histogram.get_value_at_percentile(25); //115
histogram.get_value_at_percentile(99.9); //578

Min, max, mean, standard deviation

histogram.record_value(45);
histogram.record_value(115);
histogram.record_value(198);
histogram.record_value(215);
histogram.record_value(320);
histogram.record_value(578);

histogram.get_min_value(); // 45
histogram.get_max_value(); // 578

histogram.get_mean(); //245.1666666667

histogram.get_std_deviation(); // 171.523970

Record counts

histogram.record_value(45);
histogram.record_value(115);
histogram.record_value(198);
histogram.record_value(215);
histogram.record_value(320);
histogram.record_value(578);

histogram.get_total_count(); // 6

Encoding / decoding

A histogram can either be encoded as a GLib.ByteArray, for further processing on the same machine, or serialized as base64 string. Both outputs can be compressed via ZLib. Encoding and compression are lossless.

Encoding

histogram.encode(); // Returns a base64 string representation of the histogram
histogram.encode_compressed(int compression_level = -1) // Returns a base64 string representation of a compressed histogram

histogram.encode_into_byte_buffer(); // Returns a non-compressed GLib.ByteArray representation of the histogram
histogram.encode_into_compressed_byte_buffer(int compression_level = -1)  // Returns a compressed GLib.ByteArray representation of the histogram

Decoding

Decoding an histgram will create a clone of the original. Decoding a histogram of a greater bucket size in a smaller one may fails if bbucket counts can't be stored (ex: Decoding an Histogram (int64) in a Int16Histogram).

Int16Histogram.decode(histogram); // Decode the serialized histogram in a new Int16Histogram
Int32Histogram.decode_compressed(histogram); // Decode the serialized compressed histogram in a fresh Int32Histogram
Histogram.decode_from_byte_buffer(buffer); Decode the byte array representation in a new Histogram
Histogram.decode_from_compressed_byte_buffer(compressed_buffer); // Decode the compressed byte array representation in a new Histogram

Dump in a FileStream

histogram.output_percentile_distribution(output_stream, 5, 1);

Coordinated omissions

Record with coordinated omissions :

histogram.record_value_with_expected_interval(42, 5);

Post-processing of coordinated omissions :

var new_histogram = histogram.copy_corrected_for_coordinated_omission(5);``
histogram.add_while_correcting_for_coordinated_omission(other_histogram, 5)

Iterate over recorded values

var recorded_values = histogram.recorded_values();
recorded_values.reset();
while (recorded_values.has_next()) {
    var recorded_value = recorded_values.next();
    var value = recorded_value.get_value_iterated_to();
    var count = recorded_value.get_count_at_value_iterated_to();
    var percentile = recorded_value.get_percentile();
};

Dones, TODO and propably won't be done

• Record value
• Auto resize
• Reset
• Get mean
• Get std deviation
• Get percentiles
• Output percentile distribution
• Handle concurrent accesses in RecordedValuesIterator
• Encode histogram
• Encode & compress histogram
• Decode uncompressed histograms
• Decode compressed histograms
• Corrected coordinated omissions
• Add histograms
• Substract histograms
• Iterate over percentiles
• Iterate RecordedValues
• Tags
• uint8 histogram
• uint16 histogram
• uint32 histogram
• Decode all kinds of histograms
• Packed histogram

Not likely to be done :

• Java AtomicLong > Vala counterpart
• Log reader
• Log writer