Skip to content

Latest commit

 

History

History
193 lines (149 loc) · 30.9 KB

README.md

File metadata and controls

193 lines (149 loc) · 30.9 KB

Pika Metric Exporter

Prometheus exporter for nosql Qihoo360/pika metrics.

Pika-Exporter is based on Redis-Exporter

Buiding

Build and run locally:

To start using pika_exporter, install Go and run go get

$ go get github.com/OpenAtomFoundation/pika/tools/pika_exporter
$ cd $GOPATH/src/github.com/OpenAtomFoundation/pika/tools/pika_exporter
$ make
$ ./bin/pika_exporter <flags>

For example:

$ nohup ./bin/pika_exporter -pika.addr 127.0.0.1:9221 &

Prometheus Configuration:

Add a block to the scrape_configs of your prometheus.yml config file:

scrape_configs:
  - job_name: pika
    scrape_interval: 15s
    static_configs:
      - targets: ['127.0.0.1:9121']
        labels:
          group: 'test'

Run prometheus:

prometheus --config.file=./grafana/prometheus.yml

Flags

Name Environment Variables Default Description Example
pika.host-file PIKA_HOST_FILE Path to file containing one or more pika nodes, separated by newline. NOTE: mutually exclusive with pika.addr.Each line can optionally be comma-separated with the fields <addr>,<password>,<alias>. See here for an example file. --pika.host-file ./pika_hosts_file.txt
pika.addr PIKA_ADDR Address of one or more pika nodes, separated by comma. --pika.addr 192.168.1.2:9221,192.168.1.3:9221
pika.password PIKA_PASSWORD Password for one or more pika nodes, separated by comma. --pika.password 123.com,123.com
pika.alias PIKA_ALIAS Pika instance alias for one or more pika nodes, separated by comma. --pika.alias a,b
namespace PIKA_EXPORTER_NAMESPACE pika Namespace for metrics --namespace pika
keyspace-stats-clock PIKA_EXPORTER_KEYSPACE_STATS_CLOCK -1 Stats the number of keys at keyspace-stats-clock o'clock every day, in the range [0, 23]. If < 0, not open this feature. --keyspace-stats-clock 0
check.key-patterns PIKA_EXPORTER_CHECK_KEY_PARTTERNS Comma separated list of key-patterns to export value and length/size, searched for with SCAN. --check.key-patterns db0=test*,db0=abc
check.keys PIKA_EXPORTER_CHECK_KEYS Comma separated list of keys to export value and length/size. --check.keys abc,test,wasd
check.scan-count PIKA_EXPORTER_CHECK_SCAN_COUNT 100 When check keys and executing SCAN command, scan-count assigned to COUNT. --check.scan-count 200
web.listen-address PIKA_EXPORTER_WEB_LISTEN_ADDRESS :9121 Address to listen on for web interface and telemetry. --web.listen-address ":9121"
web.telemetry-path PIKA_EXPORTER_WEB_TELEMETRY_PATH /metrics Path under which to expose metrics. --web.telemetry-path "/metrics"
log.level PIKA_EXPORTER_LOG_LEVEL info Log level, valid options:panic fatal error warn warning info debug. --log.level "debug"
log.format PIKA_EXPORTER_LOG_FORMAT json Log format, valid options:txt json. --log.format "json"
version false Show version information and exit. --version

Pika Metrics Definition

Pika Server Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_build_info Gauge {addr="", alias="", "os"="", "arch_bits"="", "pika_version"="", "pika_git_sha"="","pika_build_compile_date"=""} 1 pika binary file build info
namespace_server_info Gauge {addr="", alias="", "process_id"="", "tcp_port"="", "config_file"="", "server_id"="", "role"=""} 1 pika instance's info, the label role is the role in replication info
namespace_uptime_in_seconds Gauge {addr="", alias=""} the value of uptime_in_seconds pika instance's uptime in seconds
namespace_thread_num Gauge {addr="", alias=""} the value of thread_num pika instance's thread num
namespace_sync_thread_num Gauge {addr="", alias=""} the value of sync_thread_num pika instance's thread num for syncing

Pika Data Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_db_size Gauge {addr="", alias="", "compression"=""} the value of db_size total db data size (in bytes) of the pika instance, statistics of all files under the configured db-path
namespace_log_size Gauge {addr="", alias=""} the value of log_size total log data size (in bytes) of the pika instance, statistics of all files under the configured log-path witch contains INFO, WARNING, ERROR logs and binlog (write2fine) files for synchronization
namespace_used_memory Gauge {addr="", alias=""} the value of used_memory total used memory size (in bytes) of the pika instance
namespace_db_memtable_usage Gauge {addr="", alias=""} the value of db_memtable_usage total memtable used memory size (in bytes) of the pika instance
namespace_db_tablereader_usage Gauge {addr="", alias=""} the value of db_tablereader_usage total tablereader used memory size (in bytes) of the pika instance
(new)namespace_db_fatal Gauge {addr="", alias=""} the value of db_fatal the metrics value: 1 means errors occurred, 0 means no error

Pika Clients Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_connected_clients Gauge {addr="", alias=""} the value of connected_clients total count of connected clients in pika instance

Pika Stats Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_total_connections_received Counter {addr="", alias=""} the value of total_connections_received total count of received connections from clients in pika instance
namespace_instantaneous_ops_per_sec Gauge {addr="", alias=""} the value of instantaneous_ops_per_sec the count of prcessed operations in per seconds by pika instance
namespace_total_commands_processed Counter {addr="", alias=""} the value of total_commands_processed total count of processed commands in pika instance
namespace_is_bgsaving Gauge {addr="", alias=""} 0 or 1 the metrics value: 1 means bgsave is in progress, 0 means bgsave is not in progress
namespace_is_scaning_keyspace Gauge {addr="", alias=""} 0 or 1 the metrics value: 1 means the keyspace is scanning, 0 means not scanning
namespace_compact Gauge {addr="", alias="", compact_cron"="", "compact_interval":""} 0 or 1 the metrics value: 1 means compact is in progress, 0 means compact is not in progress

Pika Command Exec Count Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_command_exec_count Counter {addr="", alias="", "command"=""} the value of the command executed count the count of each command executed in pika instance

Pika CPU Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_used_cpu_user_children Counter {addr="", alias=""} the value of used_cpu_user_children total user CPU usage time (in seconds) of pika children instance
namespace_used_cpu_user Counter {addr="", alias=""} the value of used_cpu_user total user CPU usage time (in seconds) of pika instance
namespace_used_cpu_sys_children Counter {addr="", alias=""} the value of used_cpu_sys_children total system CPU usage time (in seconds) of pika children instance
namespace_used_cpu_sys Counter {addr="", alias=""} the value of used_cpu_sys total system CPU usage time (in seconds) of pika instance

Pika Replication Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
namespace_connected_slaves Gauge {addr="", alias=""} the value of connected_slaves the count of connected slaves, when pika instance's role is master
(no exists)namespace_partition_slave_lag Gauge {addr="", alias="", "slave_conn_fd"="", slave_ip"="", "slave_port"="", "partition"=""} parse master slave info's lag the binlog lag of all slaves of the pika instance
namespace_master_link_status Gauge {addr="", alias="", "master_host"="", "master_port"=""} 0 or 1 connection state between slave and master(1 means all partitions sync ok), when pika instance's role is slave
namespace_slave_read_only Gauge {addr="", alias="", "master_host"="", "master_port"=""} 0 or 1 is slave read only, when pika instance's role is slave
namespace_slave_priority Gauge {addr="", alias="", "master_host"="", "master_port"=""} the value of slave_priority slave priority, when pika instance's role is slave
(no exists)namespace_partition_repl_state Gauge {addr="", alias="", "master_host"="", "master_port"="", "partition"="", "repl_state"=""} 0 sync connection state between slave and master for each partition, when pika instance's role is slave
(no exists)namespace_db_binlog_offset_filenum Gauge {addr="", alias="", "db"=""} the value of binlog_offset filenum each db binlog file num for each db
(no exists)namespace_db_binlog_offset Gauge {addr="", alias="", "db"="", "safety_purge"=""} the value of binlog_offset offset each db binlog offset for each db
(new)namespace_db_consensus_last_log Gauge {addr="", alias="", "db"="", "last_log"=""} the value of consensus last_log each db consensus last_log for each db when consensus-level is enabled

Pika Keyspace Info

Metrics Name Metric Type Labels Metrics Value Metric Desc
(new)namespace_keyspace_last_start_time Gauge {addr="", alias=""} the value of Keyspace Time convert to unix seconds the start time(unix seconds) of the last statistical keyspace
namespace_keys Gauge {addr="", alias="", "db"="", "type"=""} the value of keys total count of the key-type keys for each db
namespace_expire_keys Gauge {addr="", alias="", "db"="", "type"=""} the value of expire_keys total count of the key-type expire keys for each db
namespace_invalid_keys Gauge {addr="", alias="", "db"="", "type"=""} the value of invalid_keys total count of the key-type invalid keys for each db

Pika Command Execution Time

Rocksdb Metrics

Serial Number Metric Meaning
55 rocksdb.cfstats The all metrics of compaction_L<N>: compaction.L<N>.AvgSec Average time spent per Compact. compaction.L<N>.CompCount The number of times Compact has been accumulated. compaction.L<N>.CompMergeCPU CPU time used in compression, in seconds. compaction.L<N>.CompSec Compact cumulative time, in seconds. compaction.L<N>.CompactedFiles Number of files that have completed compact. compaction.L<N>.KeyDrop Number of keys deleted in compact. compaction.L<N>.KeyIn Number of records compared during the compaction process. compaction.L<N>.MovedGB During the compaction process, the number of bytes moved to level n+1. compaction.L<N>.NumFiles Total number of sst files. compaction.L<N>.RblobGB The size of data read from the blob file by the compaction, in GB. compaction.L<N>.ReadGB Read size in GB. compaction.L<N>.ReadMBps Read rate in MBps. compaction.L<N>.RnGB When performing compact, read the size of the current layer file in GB. compaction.L<N>.Rnp1GB When performing compact, read the size of the next level file in GB. compaction.L<N>.Score Score, the higher the score, the higher the priority. compaction.L<N>.SizeBytes Total size of SST in bytes. compaction.L<N>.WblobGB The size of the blob file written during compaction, in GB. compaction.L<N>.WnewGB WNP1- Rnp1. compaction.L<N>.WriteAmp Total bytes written to level<N+1>/(total bytes read from level<N>). compaction.L<N>.WriteGB The size of the table file written during the compaction period, in GB. compaction.L<N>.WriteMBps Data write rate in compaction.
11 rocksdb.num-immutable-mem-table Number of immutable memtables not yet flushed.
12 rocksdb.num-immutable-mem-table-flushed Number of immutable memtables that have been flushed.
13 rocksdb.mem-table-flush-pending Returns 1 if there is a pending memtable flush; otherwise returns 0.
14 rocksdb.num-running-flushes Number of currently running flush operations.
15 rocksdb.compaction-pending Returns 1 if at least one compaction operation is pending; otherwise returns 0.
16 rocksdb.num-running-compactions Number of running compactions.
17 rocksdb.background-errors Total number of background errors.
18 rocksdb.cur-size-active-mem-table Approximate size, in bytes, of the active memtable.
19 rocksdb.cur-size-all-mem-tables Total size in bytes of memtables not yet flushed, including the current active memtable and the unflushed immutable memtables.
20 rocksdb.size-all-mem-tables Total size in bytes of all memtables, including the active memtable, unflushed immutable memtables, and pinned immutable memtables.
25 rocksdb.estimate-num-keys Estimated number of keys in active memtable, unflushed immutable memtables, and flushed SST files.
26 rocksdb.estimate-table-readers-mem Estimated memory size used for reading SST files, excluding block cache (such as filter and index blocks).
28 rocksdb.num-snapshots Number of unreleased snapshots in the database.
31 rocksdb.num-live-versions Number of current versions. More current versions usually indicate more SST files being used by iterators or incomplete compactions.
32 rocksdb.current-super-version-number Current number of the LSM version. It is a uint64_t integer that increments after any changes in the LSM tree. This number is not preserved after restarting the database and starts from 0 after a database restart.
33 rocksdb.estimate-live-data-size Estimated size of the activity data in bytes.For BlobDB, it also includes the actual live bytes in the version's blob file.
36 rocksdb.total-sst-files-size Total size (in bytes) of all SST files.Note: If there are too many files, it may slow down the online query.
37 rocksdb.live-sst-files-size Total size in bytes of all SST files belonging to the latest LSM tree.
47 rocksdb.block-cache-capacity The capacity of the block cache.
48 rocksdb.block-cache-usage Memory size occupied by entries in the block cache.
49 rocksdb.block-cache-pinned-usage Memory size occupied by pinned entries in the block cache.
51 rocksdb.num-blob-files The number of blob files in the current version.
52 rocksdb.blob-stats The total and size of all blob files, and the total amount of garbage (in bytes) in blob files in the current version.
53 rocksdb.total-blob-file-size The total size of all blob files across all versions.
54 rocksdb.live-blob-file-size The total size of all blob files in the current version.

Grafana Dashboard

See here

Screenshots: Overview Overview

BaseInfo

Replication

TimeConsumingOperation

KeysMetrics

RocksDB