Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Create a metrics view #1220

Open
hacdias opened this issue Oct 2, 2019 · 8 comments
Open

Feature: Create a metrics view #1220

hacdias opened this issue Oct 2, 2019 · 8 comments
Labels
area/screen/metrics Issues relating to the metrics screen effort/days Estimated to take multiple days, but less than a week exp/intermediate Prior experience is likely helpful kind/discussion Topical discussion; usually not changes to codebase kind/enhancement A net-new feature or improvement to an existing feature need/analysis Needs further analysis before proceeding need/community-input Needs input from the wider community need/maintainers-input Needs input from the current maintainer(s) P3 Low: Not priority right now status/deferred Conscious decision to pause or backlog topic/design-front-end Front-end implementation of UX/UI work topic/design-ux UX strategy, research, not solely visual design

Comments

@hacdias
Copy link
Member

hacdias commented Oct 2, 2019

We want to display a metrics view/page in the IPFS desktop app and webui. Ideally, this view would allow users to view any/all metrics they need, without having to access the CLI or other tools. This view would not have tools or complicated features that would allow users to process those metrics. Instead, they would want to use the Diagnostics View.


Original issue description

@olizilla suggested we could add a metrics tab somewhere in the Web UI where we could show the data from ${apiAddr}/debug/metrics/prometheus graphically. That would certainly be an interesting idea and useful for some kinds of users.

Leaving this as a WIP Issue


Agreed upon metrics

These metrics are ones that we've decided we will implement.

Metric Where do we get it Requires change to Kubo/libp2p/etc? Code sample Notes

Possible metrics

These metrics have strong arguments and use-cases and need to be discussed to decide whether they are useful enough to surface

Metric Where do we get it Requires change to Kubo/libp2p/etc? Code sample Notes
downloadedSize stats.bitswap No const { dataReceived } = await getIpfs().stats.bitswap(); Discussion at #1942
sharedSize stats.bitswap No const { dataSent } = await getIpfs().stats.bitswap(); Discussion at #1942
dialable kubo client/server mode Yes TBD This metric is a surfacing of whether a node is in server/client (serving/leeching) mode. We should be able to infer this, but needs analysis.

Disqualified metrics

These metrics will not be included in the metrics view for one reason or another.

Metric Where do we get it Code sample Notes

Looking for community and IPFS Implementers' feedback on this issue.

Here are some questions to help get the ideas flowing.

Which metrics should we focus on?

  • What metrics do you currently use/view/monitor and how do you obtain them?
    • Would it benefit you to see those metrics graphed in the Webui/desktop app?
    • Which of those metrics are difficult to get access to or discover?
      • i.e. Metrics that require comprehensive calculations, metrics that you seem to always forget how to obtain, metrics that require a complicated process to obtain, etc.
    • What metrics do you desire but are not currently obtainable? Why do you need them (usecase)?
      • Local node uptime/downtime - To monitor stability of my node because I need my home ipfs node to always be avail
      • Local node activity (global reqs, responses, latency, etc..) - To monitor how much my node is used (just curious)?
      • Total peers connected over time - To monitor the health of my ipfs network
      • count and frequency of requests that timeout - To denylist/allowlist certain good/bad peers?
      • etc..

What/Why/How

  • What problems do you currently have that a specific metrics view in the webui and IPFS desktop could help you solve?
  • Do you need customizable metrics views beyond selecting a time window?
  • What kind of charts & graphs would benefit you most?
  • Would you use the metrics view instead of your current tool of choice if the metrics were available in the IPFS desktop and webui?
@jessicaschilling jessicaschilling changed the title Metrics Add metrics debug info to UI Apr 7, 2020
@jessicaschilling jessicaschilling added exp/intermediate Prior experience is likely helpful effort/days Estimated to take multiple days, but less than a week kind/enhancement A net-new feature or improvement to an existing feature need/analysis Needs further analysis before proceeding P3 Low: Not priority right now status/deferred Conscious decision to pause or backlog topic/design-front-end Front-end implementation of UX/UI work topic/design-ux UX strategy, research, not solely visual design labels Apr 7, 2020
@SgtPooki SgtPooki self-assigned this Apr 21, 2022
@SgtPooki
Copy link
Member

every go-ipfs/ipfs-desktop exposes them at http://localhost:5001/debug/metrics/prometheus
some of them appear only after initial activity happens (gateway request, tcp/quic traffic, storage operation etc) so initial list may be bit empty - @lidel

@SgtPooki SgtPooki changed the title Add metrics debug info to UI Feature: Create a diagnostics & metrics view Apr 21, 2022
@SgtPooki SgtPooki added the kind/discussion Topical discussion; usually not changes to codebase label Apr 21, 2022
@SgtPooki
Copy link
Member

SgtPooki commented Apr 21, 2022

@olizilla @hacdias @lidel I'm probably going to be taking this on, so I would love to get a few ideas from those who are interested in such a view.

  1. What questions are we trying to answer with this view?
  2. What problems do we expect this view to solve, or assist with solving?

After loading up the debug/metrics/prometheus route, I was provided with the lovely wall of text below.

Click to expand
# HELP flatfs_datastore_batchcommit_errors_total Number of errored Batch.Commit calls
# TYPE flatfs_datastore_batchcommit_errors_total counter
flatfs_datastore_batchcommit_errors_total 0
# HELP flatfs_datastore_batchcommit_latency_seconds Latency distribution of Batch.Commit calls
# TYPE flatfs_datastore_batchcommit_latency_seconds histogram
flatfs_datastore_batchcommit_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_batchcommit_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_batchcommit_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_batchcommit_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_batchcommit_latency_seconds_bucket{le="1"} 0
flatfs_datastore_batchcommit_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_batchcommit_latency_seconds_sum 0
flatfs_datastore_batchcommit_latency_seconds_count 0
# HELP flatfs_datastore_batchcommit_total Total number of Batch.Commit calls
# TYPE flatfs_datastore_batchcommit_total counter
flatfs_datastore_batchcommit_total 0
# HELP flatfs_datastore_batchdelete_errors_total Number of errored Batch.Delete calls
# TYPE flatfs_datastore_batchdelete_errors_total counter
flatfs_datastore_batchdelete_errors_total 0
# HELP flatfs_datastore_batchdelete_latency_seconds Latency distribution of Batch.Delete calls
# TYPE flatfs_datastore_batchdelete_latency_seconds histogram
flatfs_datastore_batchdelete_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_batchdelete_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_batchdelete_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_batchdelete_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_batchdelete_latency_seconds_bucket{le="1"} 0
flatfs_datastore_batchdelete_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_batchdelete_latency_seconds_sum 0
flatfs_datastore_batchdelete_latency_seconds_count 0
# HELP flatfs_datastore_batchdelete_total Total number of Batch.Delete calls
# TYPE flatfs_datastore_batchdelete_total counter
flatfs_datastore_batchdelete_total 0
# HELP flatfs_datastore_batchput_errors_total Number of errored Batch.Put calls
# TYPE flatfs_datastore_batchput_errors_total counter
flatfs_datastore_batchput_errors_total 0
# HELP flatfs_datastore_batchput_latency_seconds Latency distribution of Batch.Put calls
# TYPE flatfs_datastore_batchput_latency_seconds histogram
flatfs_datastore_batchput_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_batchput_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_batchput_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_batchput_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_batchput_latency_seconds_bucket{le="1"} 0
flatfs_datastore_batchput_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_batchput_latency_seconds_sum 0
flatfs_datastore_batchput_latency_seconds_count 0
# HELP flatfs_datastore_batchput_size_bytes Size distribution of byte slices put into batches
# TYPE flatfs_datastore_batchput_size_bytes histogram
flatfs_datastore_batchput_size_bytes_bucket{le="64"} 0
flatfs_datastore_batchput_size_bytes_bucket{le="4096"} 0
flatfs_datastore_batchput_size_bytes_bucket{le="262144"} 0
flatfs_datastore_batchput_size_bytes_bucket{le="1.6777216e+07"} 0
flatfs_datastore_batchput_size_bytes_bucket{le="+Inf"} 0
flatfs_datastore_batchput_size_bytes_sum 0
flatfs_datastore_batchput_size_bytes_count 0
# HELP flatfs_datastore_batchput_total Total number of Batch.Put calls
# TYPE flatfs_datastore_batchput_total counter
flatfs_datastore_batchput_total 0
# HELP flatfs_datastore_check_errors_total Number of errored Datastore.Check calls
# TYPE flatfs_datastore_check_errors_total counter
flatfs_datastore_check_errors_total 0
# HELP flatfs_datastore_check_latency_seconds Latency distribution of Datastore.Check calls
# TYPE flatfs_datastore_check_latency_seconds histogram
flatfs_datastore_check_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_check_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_check_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_check_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_check_latency_seconds_bucket{le="1"} 0
flatfs_datastore_check_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_check_latency_seconds_sum 0
flatfs_datastore_check_latency_seconds_count 0
# HELP flatfs_datastore_check_total Total number of Datastore.Check calls
# TYPE flatfs_datastore_check_total counter
flatfs_datastore_check_total 0
# HELP flatfs_datastore_delete_errors_total Number of errored Datastore.Delete calls
# TYPE flatfs_datastore_delete_errors_total counter
flatfs_datastore_delete_errors_total 0
# HELP flatfs_datastore_delete_latency_seconds Latency distribution of Datastore.Delete calls
# TYPE flatfs_datastore_delete_latency_seconds histogram
flatfs_datastore_delete_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_delete_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_delete_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_delete_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_delete_latency_seconds_bucket{le="1"} 0
flatfs_datastore_delete_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_delete_latency_seconds_sum 0
flatfs_datastore_delete_latency_seconds_count 0
# HELP flatfs_datastore_delete_total Total number of Datastore.Delete calls
# TYPE flatfs_datastore_delete_total counter
flatfs_datastore_delete_total 0
# HELP flatfs_datastore_du_errors_total Number of errored Datastore.DiskUsage calls
# TYPE flatfs_datastore_du_errors_total counter
flatfs_datastore_du_errors_total 0
# HELP flatfs_datastore_du_latency_seconds Latency distribution of Datastore.DiskUsage calls
# TYPE flatfs_datastore_du_latency_seconds histogram
flatfs_datastore_du_latency_seconds_bucket{le="0.0001"} 2
flatfs_datastore_du_latency_seconds_bucket{le="0.001"} 2
flatfs_datastore_du_latency_seconds_bucket{le="0.01"} 2
flatfs_datastore_du_latency_seconds_bucket{le="0.1"} 2
flatfs_datastore_du_latency_seconds_bucket{le="1"} 2
flatfs_datastore_du_latency_seconds_bucket{le="+Inf"} 2
flatfs_datastore_du_latency_seconds_sum 3.7499999999999997e-06
flatfs_datastore_du_latency_seconds_count 2
# HELP flatfs_datastore_du_total Total number of Datastore.DiskUsage calls
# TYPE flatfs_datastore_du_total counter
flatfs_datastore_du_total 2
# HELP flatfs_datastore_gc_errors_total Number of errored Datastore.CollectGarbage calls
# TYPE flatfs_datastore_gc_errors_total counter
flatfs_datastore_gc_errors_total 0
# HELP flatfs_datastore_gc_latency_seconds Latency distribution of Datastore.CollectGarbage calls
# TYPE flatfs_datastore_gc_latency_seconds histogram
flatfs_datastore_gc_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_gc_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_gc_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_gc_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_gc_latency_seconds_bucket{le="1"} 0
flatfs_datastore_gc_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_gc_latency_seconds_sum 0
flatfs_datastore_gc_latency_seconds_count 0
# HELP flatfs_datastore_gc_total Total number of Datastore.CollectGarbage calls
# TYPE flatfs_datastore_gc_total counter
flatfs_datastore_gc_total 0
# HELP flatfs_datastore_get_errors_total Number of errored Datastore.Get calls
# TYPE flatfs_datastore_get_errors_total counter
flatfs_datastore_get_errors_total 0
# HELP flatfs_datastore_get_latency_seconds Latency distribution of Datastore.Get calls
# TYPE flatfs_datastore_get_latency_seconds histogram
flatfs_datastore_get_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_get_latency_seconds_bucket{le="0.001"} 1
flatfs_datastore_get_latency_seconds_bucket{le="0.01"} 1
flatfs_datastore_get_latency_seconds_bucket{le="0.1"} 1
flatfs_datastore_get_latency_seconds_bucket{le="1"} 1
flatfs_datastore_get_latency_seconds_bucket{le="+Inf"} 1
flatfs_datastore_get_latency_seconds_sum 0.000946375
flatfs_datastore_get_latency_seconds_count 1
# HELP flatfs_datastore_get_size_bytes Size distribution of retrieved byte slices
# TYPE flatfs_datastore_get_size_bytes histogram
flatfs_datastore_get_size_bytes_bucket{le="64"} 0
flatfs_datastore_get_size_bytes_bucket{le="4096"} 1
flatfs_datastore_get_size_bytes_bucket{le="262144"} 1
flatfs_datastore_get_size_bytes_bucket{le="1.6777216e+07"} 1
flatfs_datastore_get_size_bytes_bucket{le="+Inf"} 1
flatfs_datastore_get_size_bytes_sum 69
flatfs_datastore_get_size_bytes_count 1
# HELP flatfs_datastore_get_total Total number of Datastore.Get calls
# TYPE flatfs_datastore_get_total counter
flatfs_datastore_get_total 1
# HELP flatfs_datastore_getsize_errors_total Number of errored Datastore.GetSize calls
# TYPE flatfs_datastore_getsize_errors_total counter
flatfs_datastore_getsize_errors_total 0
# HELP flatfs_datastore_getsize_latency_seconds Latency distribution of Datastore.GetSize calls
# TYPE flatfs_datastore_getsize_latency_seconds histogram
flatfs_datastore_getsize_latency_seconds_bucket{le="0.0001"} 1241
flatfs_datastore_getsize_latency_seconds_bucket{le="0.001"} 1558
flatfs_datastore_getsize_latency_seconds_bucket{le="0.01"} 1564
flatfs_datastore_getsize_latency_seconds_bucket{le="0.1"} 1564
flatfs_datastore_getsize_latency_seconds_bucket{le="1"} 1564
flatfs_datastore_getsize_latency_seconds_bucket{le="+Inf"} 1564
flatfs_datastore_getsize_latency_seconds_sum 0.11992588499999995
flatfs_datastore_getsize_latency_seconds_count 1564
# HELP flatfs_datastore_getsize_total Total number of Datastore.GetSize calls
# TYPE flatfs_datastore_getsize_total counter
flatfs_datastore_getsize_total 1564
# HELP flatfs_datastore_has_errors_total Number of errored Datastore.Has calls
# TYPE flatfs_datastore_has_errors_total counter
flatfs_datastore_has_errors_total 0
# HELP flatfs_datastore_has_latency_seconds Latency distribution of Datastore.Has calls
# TYPE flatfs_datastore_has_latency_seconds histogram
flatfs_datastore_has_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_has_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_has_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_has_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_has_latency_seconds_bucket{le="1"} 0
flatfs_datastore_has_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_has_latency_seconds_sum 0
flatfs_datastore_has_latency_seconds_count 0
# HELP flatfs_datastore_has_total Total number of Datastore.Has calls
# TYPE flatfs_datastore_has_total counter
flatfs_datastore_has_total 0
# HELP flatfs_datastore_put_errors_total Number of errored Datastore.Put calls
# TYPE flatfs_datastore_put_errors_total counter
flatfs_datastore_put_errors_total 0
# HELP flatfs_datastore_put_latency_seconds Latency distribution of Datastore.Put calls
# TYPE flatfs_datastore_put_latency_seconds histogram
flatfs_datastore_put_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_put_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_put_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_put_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_put_latency_seconds_bucket{le="1"} 0
flatfs_datastore_put_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_put_latency_seconds_sum 0
flatfs_datastore_put_latency_seconds_count 0
# HELP flatfs_datastore_put_size_bytes Size distribution of stored byte slices
# TYPE flatfs_datastore_put_size_bytes histogram
flatfs_datastore_put_size_bytes_bucket{le="64"} 0
flatfs_datastore_put_size_bytes_bucket{le="4096"} 0
flatfs_datastore_put_size_bytes_bucket{le="262144"} 0
flatfs_datastore_put_size_bytes_bucket{le="1.6777216e+07"} 0
flatfs_datastore_put_size_bytes_bucket{le="+Inf"} 0
flatfs_datastore_put_size_bytes_sum 0
flatfs_datastore_put_size_bytes_count 0
# HELP flatfs_datastore_put_total Total number of Datastore.Put calls
# TYPE flatfs_datastore_put_total counter
flatfs_datastore_put_total 0
# HELP flatfs_datastore_query_errors_total Number of errored Datastore.Query calls
# TYPE flatfs_datastore_query_errors_total counter
flatfs_datastore_query_errors_total 0
# HELP flatfs_datastore_query_latency_seconds Latency distribution of Datastore.Query calls
# TYPE flatfs_datastore_query_latency_seconds histogram
flatfs_datastore_query_latency_seconds_bucket{le="0.0001"} 2
flatfs_datastore_query_latency_seconds_bucket{le="0.001"} 3
flatfs_datastore_query_latency_seconds_bucket{le="0.01"} 3
flatfs_datastore_query_latency_seconds_bucket{le="0.1"} 3
flatfs_datastore_query_latency_seconds_bucket{le="1"} 3
flatfs_datastore_query_latency_seconds_bucket{le="+Inf"} 3
flatfs_datastore_query_latency_seconds_sum 0.00036225
flatfs_datastore_query_latency_seconds_count 3
# HELP flatfs_datastore_query_total Total number of Datastore.Query calls
# TYPE flatfs_datastore_query_total counter
flatfs_datastore_query_total 3
# HELP flatfs_datastore_scrub_errors_total Number of errored Datastore.Scrub calls
# TYPE flatfs_datastore_scrub_errors_total counter
flatfs_datastore_scrub_errors_total 0
# HELP flatfs_datastore_scrub_latency_seconds Latency distribution of Datastore.Scrub calls
# TYPE flatfs_datastore_scrub_latency_seconds histogram
flatfs_datastore_scrub_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="1"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_scrub_latency_seconds_sum 0
flatfs_datastore_scrub_latency_seconds_count 0
# HELP flatfs_datastore_scrub_total Total number of Datastore.Scrub calls
# TYPE flatfs_datastore_scrub_total counter
flatfs_datastore_scrub_total 0
# HELP flatfs_datastore_sync_errors_total Number of errored Datastore.Sync calls
# TYPE flatfs_datastore_sync_errors_total counter
flatfs_datastore_sync_errors_total 0
# HELP flatfs_datastore_sync_latency_seconds Latency distribution of Datastore.Sync calls
# TYPE flatfs_datastore_sync_latency_seconds histogram
flatfs_datastore_sync_latency_seconds_bucket{le="0.0001"} 0
flatfs_datastore_sync_latency_seconds_bucket{le="0.001"} 0
flatfs_datastore_sync_latency_seconds_bucket{le="0.01"} 0
flatfs_datastore_sync_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_sync_latency_seconds_bucket{le="1"} 0
flatfs_datastore_sync_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_sync_latency_seconds_sum 0
flatfs_datastore_sync_latency_seconds_count 0
# HELP flatfs_datastore_sync_total Total number of Datastore.Sync calls
# TYPE flatfs_datastore_sync_total counter
flatfs_datastore_sync_total 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 6.2917e-05
go_gc_duration_seconds{quantile="0.25"} 8.3999e-05
go_gc_duration_seconds{quantile="0.5"} 0.000206667
go_gc_duration_seconds{quantile="0.75"} 0.000530583
go_gc_duration_seconds{quantile="1"} 0.001019
go_gc_duration_seconds_sum 0.003010501
go_gc_duration_seconds_count 10
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 2095
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.16.12"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 8.8467e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 2.993212e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.542692e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 3.015127e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0005584668913431851
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 8.73596e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 8.8467e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.4010752e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.00999168e+08
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 1.145842e+06
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 1.8243584e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.2500992e+08
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.650581604131364e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4.160969e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 12000
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 1.444456e+06
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 1.540096e+06
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.11606688e+08
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.259804e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 9.207808e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 9.207808e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.47312664e+08
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 16
# HELP ipfs_bitswap_active_block_tasks Total number of active blockstore tasks
# TYPE ipfs_bitswap_active_block_tasks gauge
ipfs_bitswap_active_block_tasks 0
# HELP ipfs_bitswap_active_tasks Total number of active tasks
# TYPE ipfs_bitswap_active_tasks gauge
ipfs_bitswap_active_tasks 0
# HELP ipfs_bitswap_pending_block_tasks Total number of pending blockstore tasks
# TYPE ipfs_bitswap_pending_block_tasks gauge
ipfs_bitswap_pending_block_tasks 0
# HELP ipfs_bitswap_pending_tasks Total number of pending tasks
# TYPE ipfs_bitswap_pending_tasks gauge
ipfs_bitswap_pending_tasks 0
# HELP ipfs_bitswap_recv_all_blocks_bytes Summary of all data blocks recived
# TYPE ipfs_bitswap_recv_all_blocks_bytes histogram
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="64"} 0
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="1024"} 0
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="16384"} 0
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="262144"} 0
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="262159"} 0
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="4.194304e+06"} 0
ipfs_bitswap_recv_all_blocks_bytes_bucket{le="+Inf"} 0
ipfs_bitswap_recv_all_blocks_bytes_sum 0
ipfs_bitswap_recv_all_blocks_bytes_count 0
# HELP ipfs_bitswap_recv_dup_blocks_bytes Summary of duplicate data blocks recived
# TYPE ipfs_bitswap_recv_dup_blocks_bytes histogram
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="64"} 0
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="1024"} 0
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="16384"} 0
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="262144"} 0
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="262159"} 0
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="4.194304e+06"} 0
ipfs_bitswap_recv_dup_blocks_bytes_bucket{le="+Inf"} 0
ipfs_bitswap_recv_dup_blocks_bytes_sum 0
ipfs_bitswap_recv_dup_blocks_bytes_count 0
# HELP ipfs_bitswap_send_times Histogram of how long it takes to send messages in this bitswap
# TYPE ipfs_bitswap_send_times histogram
ipfs_bitswap_send_times_bucket{le="1"} 0
ipfs_bitswap_send_times_bucket{le="10"} 0
ipfs_bitswap_send_times_bucket{le="30"} 0
ipfs_bitswap_send_times_bucket{le="60"} 0
ipfs_bitswap_send_times_bucket{le="90"} 0
ipfs_bitswap_send_times_bucket{le="120"} 0
ipfs_bitswap_send_times_bucket{le="600"} 0
ipfs_bitswap_send_times_bucket{le="+Inf"} 0
ipfs_bitswap_send_times_sum 0
ipfs_bitswap_send_times_count 0
# HELP ipfs_bitswap_sent_all_blocks_bytes Histogram of blocks sent by this bitswap
# TYPE ipfs_bitswap_sent_all_blocks_bytes histogram
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="64"} 0
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="1024"} 0
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="16384"} 0
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="262144"} 0
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="262159"} 0
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="4.194304e+06"} 0
ipfs_bitswap_sent_all_blocks_bytes_bucket{le="+Inf"} 0
ipfs_bitswap_sent_all_blocks_bytes_sum 0
ipfs_bitswap_sent_all_blocks_bytes_count 0
# HELP ipfs_bitswap_want_blocks_total Number of want-blocks in wantlist.
# TYPE ipfs_bitswap_want_blocks_total gauge
ipfs_bitswap_want_blocks_total 0
# HELP ipfs_bitswap_wantlist_total Number of items in wantlist.
# TYPE ipfs_bitswap_wantlist_total gauge
ipfs_bitswap_wantlist_total 0
# HELP ipfs_bs_cache_arc_hits_total Number of ARC cache hits
# TYPE ipfs_bs_cache_arc_hits_total counter
ipfs_bs_cache_arc_hits_total 893
# HELP ipfs_bs_cache_arc_total Total number of ARC cache requests
# TYPE ipfs_bs_cache_arc_total counter
ipfs_bs_cache_arc_total 2458
# HELP ipfs_fsrepo_datastore_batchcommit_errors_total Number of errored Batch.Commit calls
# TYPE ipfs_fsrepo_datastore_batchcommit_errors_total counter
ipfs_fsrepo_datastore_batchcommit_errors_total 0
# HELP ipfs_fsrepo_datastore_batchcommit_latency_seconds Latency distribution of Batch.Commit calls
# TYPE ipfs_fsrepo_datastore_batchcommit_latency_seconds histogram
ipfs_fsrepo_datastore_batchcommit_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_sum 0
ipfs_fsrepo_datastore_batchcommit_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_batchcommit_total Total number of Batch.Commit calls
# TYPE ipfs_fsrepo_datastore_batchcommit_total counter
ipfs_fsrepo_datastore_batchcommit_total 0
# HELP ipfs_fsrepo_datastore_batchdelete_errors_total Number of errored Batch.Delete calls
# TYPE ipfs_fsrepo_datastore_batchdelete_errors_total counter
ipfs_fsrepo_datastore_batchdelete_errors_total 0
# HELP ipfs_fsrepo_datastore_batchdelete_latency_seconds Latency distribution of Batch.Delete calls
# TYPE ipfs_fsrepo_datastore_batchdelete_latency_seconds histogram
ipfs_fsrepo_datastore_batchdelete_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_sum 0
ipfs_fsrepo_datastore_batchdelete_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_batchdelete_total Total number of Batch.Delete calls
# TYPE ipfs_fsrepo_datastore_batchdelete_total counter
ipfs_fsrepo_datastore_batchdelete_total 0
# HELP ipfs_fsrepo_datastore_batchput_errors_total Number of errored Batch.Put calls
# TYPE ipfs_fsrepo_datastore_batchput_errors_total counter
ipfs_fsrepo_datastore_batchput_errors_total 0
# HELP ipfs_fsrepo_datastore_batchput_latency_seconds Latency distribution of Batch.Put calls
# TYPE ipfs_fsrepo_datastore_batchput_latency_seconds histogram
ipfs_fsrepo_datastore_batchput_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_batchput_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_batchput_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_batchput_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_batchput_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_batchput_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_batchput_latency_seconds_sum 0
ipfs_fsrepo_datastore_batchput_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_batchput_size_bytes Size distribution of byte slices put into batches
# TYPE ipfs_fsrepo_datastore_batchput_size_bytes histogram
ipfs_fsrepo_datastore_batchput_size_bytes_bucket{le="64"} 0
ipfs_fsrepo_datastore_batchput_size_bytes_bucket{le="4096"} 0
ipfs_fsrepo_datastore_batchput_size_bytes_bucket{le="262144"} 0
ipfs_fsrepo_datastore_batchput_size_bytes_bucket{le="1.6777216e+07"} 0
ipfs_fsrepo_datastore_batchput_size_bytes_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_batchput_size_bytes_sum 0
ipfs_fsrepo_datastore_batchput_size_bytes_count 0
# HELP ipfs_fsrepo_datastore_batchput_total Total number of Batch.Put calls
# TYPE ipfs_fsrepo_datastore_batchput_total counter
ipfs_fsrepo_datastore_batchput_total 0
# HELP ipfs_fsrepo_datastore_check_errors_total Number of errored Datastore.Check calls
# TYPE ipfs_fsrepo_datastore_check_errors_total counter
ipfs_fsrepo_datastore_check_errors_total 0
# HELP ipfs_fsrepo_datastore_check_latency_seconds Latency distribution of Datastore.Check calls
# TYPE ipfs_fsrepo_datastore_check_latency_seconds histogram
ipfs_fsrepo_datastore_check_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_check_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_check_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_check_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_check_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_check_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_check_latency_seconds_sum 0
ipfs_fsrepo_datastore_check_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_check_total Total number of Datastore.Check calls
# TYPE ipfs_fsrepo_datastore_check_total counter
ipfs_fsrepo_datastore_check_total 0
# HELP ipfs_fsrepo_datastore_delete_errors_total Number of errored Datastore.Delete calls
# TYPE ipfs_fsrepo_datastore_delete_errors_total counter
ipfs_fsrepo_datastore_delete_errors_total 0
# HELP ipfs_fsrepo_datastore_delete_latency_seconds Latency distribution of Datastore.Delete calls
# TYPE ipfs_fsrepo_datastore_delete_latency_seconds histogram
ipfs_fsrepo_datastore_delete_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_delete_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_delete_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_delete_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_delete_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_delete_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_delete_latency_seconds_sum 0
ipfs_fsrepo_datastore_delete_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_delete_total Total number of Datastore.Delete calls
# TYPE ipfs_fsrepo_datastore_delete_total counter
ipfs_fsrepo_datastore_delete_total 0
# HELP ipfs_fsrepo_datastore_du_errors_total Number of errored Datastore.DiskUsage calls
# TYPE ipfs_fsrepo_datastore_du_errors_total counter
ipfs_fsrepo_datastore_du_errors_total 0
# HELP ipfs_fsrepo_datastore_du_latency_seconds Latency distribution of Datastore.DiskUsage calls
# TYPE ipfs_fsrepo_datastore_du_latency_seconds histogram
ipfs_fsrepo_datastore_du_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_du_latency_seconds_bucket{le="0.001"} 2
ipfs_fsrepo_datastore_du_latency_seconds_bucket{le="0.01"} 2
ipfs_fsrepo_datastore_du_latency_seconds_bucket{le="0.1"} 2
ipfs_fsrepo_datastore_du_latency_seconds_bucket{le="1"} 2
ipfs_fsrepo_datastore_du_latency_seconds_bucket{le="+Inf"} 2
ipfs_fsrepo_datastore_du_latency_seconds_sum 0.000299083
ipfs_fsrepo_datastore_du_latency_seconds_count 2
# HELP ipfs_fsrepo_datastore_du_total Total number of Datastore.DiskUsage calls
# TYPE ipfs_fsrepo_datastore_du_total counter
ipfs_fsrepo_datastore_du_total 2
# HELP ipfs_fsrepo_datastore_gc_errors_total Number of errored Datastore.CollectGarbage calls
# TYPE ipfs_fsrepo_datastore_gc_errors_total counter
ipfs_fsrepo_datastore_gc_errors_total 0
# HELP ipfs_fsrepo_datastore_gc_latency_seconds Latency distribution of Datastore.CollectGarbage calls
# TYPE ipfs_fsrepo_datastore_gc_latency_seconds histogram
ipfs_fsrepo_datastore_gc_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_gc_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_gc_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_gc_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_gc_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_gc_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_gc_latency_seconds_sum 0
ipfs_fsrepo_datastore_gc_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_gc_total Total number of Datastore.CollectGarbage calls
# TYPE ipfs_fsrepo_datastore_gc_total counter
ipfs_fsrepo_datastore_gc_total 0
# HELP ipfs_fsrepo_datastore_get_errors_total Number of errored Datastore.Get calls
# TYPE ipfs_fsrepo_datastore_get_errors_total counter
ipfs_fsrepo_datastore_get_errors_total 0
# HELP ipfs_fsrepo_datastore_get_latency_seconds Latency distribution of Datastore.Get calls
# TYPE ipfs_fsrepo_datastore_get_latency_seconds histogram
ipfs_fsrepo_datastore_get_latency_seconds_bucket{le="0.0001"} 2
ipfs_fsrepo_datastore_get_latency_seconds_bucket{le="0.001"} 4
ipfs_fsrepo_datastore_get_latency_seconds_bucket{le="0.01"} 6
ipfs_fsrepo_datastore_get_latency_seconds_bucket{le="0.1"} 6
ipfs_fsrepo_datastore_get_latency_seconds_bucket{le="1"} 6
ipfs_fsrepo_datastore_get_latency_seconds_bucket{le="+Inf"} 6
ipfs_fsrepo_datastore_get_latency_seconds_sum 0.007452710000000001
ipfs_fsrepo_datastore_get_latency_seconds_count 6
# HELP ipfs_fsrepo_datastore_get_size_bytes Size distribution of retrieved byte slices
# TYPE ipfs_fsrepo_datastore_get_size_bytes histogram
ipfs_fsrepo_datastore_get_size_bytes_bucket{le="64"} 2
ipfs_fsrepo_datastore_get_size_bytes_bucket{le="4096"} 6
ipfs_fsrepo_datastore_get_size_bytes_bucket{le="262144"} 6
ipfs_fsrepo_datastore_get_size_bytes_bucket{le="1.6777216e+07"} 6
ipfs_fsrepo_datastore_get_size_bytes_bucket{le="+Inf"} 6
ipfs_fsrepo_datastore_get_size_bytes_sum 1238
ipfs_fsrepo_datastore_get_size_bytes_count 6
# HELP ipfs_fsrepo_datastore_get_total Total number of Datastore.Get calls
# TYPE ipfs_fsrepo_datastore_get_total counter
ipfs_fsrepo_datastore_get_total 6
# HELP ipfs_fsrepo_datastore_getsize_errors_total Number of errored Datastore.GetSize calls
# TYPE ipfs_fsrepo_datastore_getsize_errors_total counter
ipfs_fsrepo_datastore_getsize_errors_total 0
# HELP ipfs_fsrepo_datastore_getsize_latency_seconds Latency distribution of Datastore.GetSize calls
# TYPE ipfs_fsrepo_datastore_getsize_latency_seconds histogram
ipfs_fsrepo_datastore_getsize_latency_seconds_bucket{le="0.0001"} 1214
ipfs_fsrepo_datastore_getsize_latency_seconds_bucket{le="0.001"} 1555
ipfs_fsrepo_datastore_getsize_latency_seconds_bucket{le="0.01"} 1564
ipfs_fsrepo_datastore_getsize_latency_seconds_bucket{le="0.1"} 1564
ipfs_fsrepo_datastore_getsize_latency_seconds_bucket{le="1"} 1564
ipfs_fsrepo_datastore_getsize_latency_seconds_bucket{le="+Inf"} 1564
ipfs_fsrepo_datastore_getsize_latency_seconds_sum 0.12957606799999982
ipfs_fsrepo_datastore_getsize_latency_seconds_count 1564
# HELP ipfs_fsrepo_datastore_getsize_total Total number of Datastore.GetSize calls
# TYPE ipfs_fsrepo_datastore_getsize_total counter
ipfs_fsrepo_datastore_getsize_total 1564
# HELP ipfs_fsrepo_datastore_has_errors_total Number of errored Datastore.Has calls
# TYPE ipfs_fsrepo_datastore_has_errors_total counter
ipfs_fsrepo_datastore_has_errors_total 0
# HELP ipfs_fsrepo_datastore_has_latency_seconds Latency distribution of Datastore.Has calls
# TYPE ipfs_fsrepo_datastore_has_latency_seconds histogram
ipfs_fsrepo_datastore_has_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_has_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_has_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_has_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_has_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_has_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_has_latency_seconds_sum 0
ipfs_fsrepo_datastore_has_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_has_total Total number of Datastore.Has calls
# TYPE ipfs_fsrepo_datastore_has_total counter
ipfs_fsrepo_datastore_has_total 0
# HELP ipfs_fsrepo_datastore_put_errors_total Number of errored Datastore.Put calls
# TYPE ipfs_fsrepo_datastore_put_errors_total counter
ipfs_fsrepo_datastore_put_errors_total 0
# HELP ipfs_fsrepo_datastore_put_latency_seconds Latency distribution of Datastore.Put calls
# TYPE ipfs_fsrepo_datastore_put_latency_seconds histogram
ipfs_fsrepo_datastore_put_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_put_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_put_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_put_latency_seconds_bucket{le="0.1"} 2
ipfs_fsrepo_datastore_put_latency_seconds_bucket{le="1"} 2
ipfs_fsrepo_datastore_put_latency_seconds_bucket{le="+Inf"} 2
ipfs_fsrepo_datastore_put_latency_seconds_sum 0.045931542
ipfs_fsrepo_datastore_put_latency_seconds_count 2
# HELP ipfs_fsrepo_datastore_put_size_bytes Size distribution of stored byte slices
# TYPE ipfs_fsrepo_datastore_put_size_bytes histogram
ipfs_fsrepo_datastore_put_size_bytes_bucket{le="64"} 0
ipfs_fsrepo_datastore_put_size_bytes_bucket{le="4096"} 2
ipfs_fsrepo_datastore_put_size_bytes_bucket{le="262144"} 2
ipfs_fsrepo_datastore_put_size_bytes_bucket{le="1.6777216e+07"} 2
ipfs_fsrepo_datastore_put_size_bytes_bucket{le="+Inf"} 2
ipfs_fsrepo_datastore_put_size_bytes_sum 782
ipfs_fsrepo_datastore_put_size_bytes_count 2
# HELP ipfs_fsrepo_datastore_put_total Total number of Datastore.Put calls
# TYPE ipfs_fsrepo_datastore_put_total counter
ipfs_fsrepo_datastore_put_total 2
# HELP ipfs_fsrepo_datastore_query_errors_total Number of errored Datastore.Query calls
# TYPE ipfs_fsrepo_datastore_query_errors_total counter
ipfs_fsrepo_datastore_query_errors_total 0
# HELP ipfs_fsrepo_datastore_query_latency_seconds Latency distribution of Datastore.Query calls
# TYPE ipfs_fsrepo_datastore_query_latency_seconds histogram
ipfs_fsrepo_datastore_query_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_query_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_query_latency_seconds_bucket{le="0.01"} 2
ipfs_fsrepo_datastore_query_latency_seconds_bucket{le="0.1"} 4
ipfs_fsrepo_datastore_query_latency_seconds_bucket{le="1"} 4
ipfs_fsrepo_datastore_query_latency_seconds_bucket{le="+Inf"} 4
ipfs_fsrepo_datastore_query_latency_seconds_sum 0.109688791
ipfs_fsrepo_datastore_query_latency_seconds_count 4
# HELP ipfs_fsrepo_datastore_query_total Total number of Datastore.Query calls
# TYPE ipfs_fsrepo_datastore_query_total counter
ipfs_fsrepo_datastore_query_total 4
# HELP ipfs_fsrepo_datastore_scrub_errors_total Number of errored Datastore.Scrub calls
# TYPE ipfs_fsrepo_datastore_scrub_errors_total counter
ipfs_fsrepo_datastore_scrub_errors_total 0
# HELP ipfs_fsrepo_datastore_scrub_latency_seconds Latency distribution of Datastore.Scrub calls
# TYPE ipfs_fsrepo_datastore_scrub_latency_seconds histogram
ipfs_fsrepo_datastore_scrub_latency_seconds_bucket{le="0.0001"} 0
ipfs_fsrepo_datastore_scrub_latency_seconds_bucket{le="0.001"} 0
ipfs_fsrepo_datastore_scrub_latency_seconds_bucket{le="0.01"} 0
ipfs_fsrepo_datastore_scrub_latency_seconds_bucket{le="0.1"} 0
ipfs_fsrepo_datastore_scrub_latency_seconds_bucket{le="1"} 0
ipfs_fsrepo_datastore_scrub_latency_seconds_bucket{le="+Inf"} 0
ipfs_fsrepo_datastore_scrub_latency_seconds_sum 0
ipfs_fsrepo_datastore_scrub_latency_seconds_count 0
# HELP ipfs_fsrepo_datastore_scrub_total Total number of Datastore.Scrub calls
# TYPE ipfs_fsrepo_datastore_scrub_total counter
ipfs_fsrepo_datastore_scrub_total 0
# HELP ipfs_fsrepo_datastore_sync_errors_total Number of errored Datastore.Sync calls
# TYPE ipfs_fsrepo_datastore_sync_errors_total counter
ipfs_fsrepo_datastore_sync_errors_total 0
# HELP ipfs_fsrepo_datastore_sync_latency_seconds Latency distribution of Datastore.Sync calls
# TYPE ipfs_fsrepo_datastore_sync_latency_seconds histogram
ipfs_fsrepo_datastore_sync_latency_seconds_bucket{le="0.0001"} 1
ipfs_fsrepo_datastore_sync_latency_seconds_bucket{le="0.001"} 1
ipfs_fsrepo_datastore_sync_latency_seconds_bucket{le="0.01"} 1
ipfs_fsrepo_datastore_sync_latency_seconds_bucket{le="0.1"} 1
ipfs_fsrepo_datastore_sync_latency_seconds_bucket{le="1"} 1
ipfs_fsrepo_datastore_sync_latency_seconds_bucket{le="+Inf"} 1
ipfs_fsrepo_datastore_sync_latency_seconds_sum 4.75e-06
ipfs_fsrepo_datastore_sync_latency_seconds_count 1
# HELP ipfs_fsrepo_datastore_sync_total Total number of Datastore.Sync calls
# TYPE ipfs_fsrepo_datastore_sync_total counter
ipfs_fsrepo_datastore_sync_total 1
# HELP ipfs_http_request_duration_seconds The HTTP request latencies in seconds.
# TYPE ipfs_http_request_duration_seconds summary
ipfs_http_request_duration_seconds{handler="api",quantile="0.5"} 0.000308208
ipfs_http_request_duration_seconds{handler="api",quantile="0.9"} 0.004977041
ipfs_http_request_duration_seconds{handler="api",quantile="0.99"} 0.127640625
ipfs_http_request_duration_seconds_sum{handler="api"} 0.203921085
ipfs_http_request_duration_seconds_count{handler="api"} 42
ipfs_http_request_duration_seconds{handler="gateway",quantile="0.5"} 0.013123375
ipfs_http_request_duration_seconds{handler="gateway",quantile="0.9"} 0.013123375
ipfs_http_request_duration_seconds{handler="gateway",quantile="0.99"} 0.013123375
ipfs_http_request_duration_seconds_sum{handler="gateway"} 0.013123375
ipfs_http_request_duration_seconds_count{handler="gateway"} 1
# HELP ipfs_http_request_size_bytes The HTTP request sizes in bytes.
# TYPE ipfs_http_request_size_bytes summary
ipfs_http_request_size_bytes{handler="api",quantile="0.5"} 238
ipfs_http_request_size_bytes{handler="api",quantile="0.9"} 270
ipfs_http_request_size_bytes{handler="api",quantile="0.99"} 543
ipfs_http_request_size_bytes_sum{handler="api"} 11173
ipfs_http_request_size_bytes_count{handler="api"} 42
ipfs_http_request_size_bytes{handler="gateway",quantile="0.5"} 303
ipfs_http_request_size_bytes{handler="gateway",quantile="0.9"} 303
ipfs_http_request_size_bytes{handler="gateway",quantile="0.99"} 303
ipfs_http_request_size_bytes_sum{handler="gateway"} 303
ipfs_http_request_size_bytes_count{handler="gateway"} 1
# HELP ipfs_http_requests_total Total number of HTTP requests made.
# TYPE ipfs_http_requests_total counter
ipfs_http_requests_total{code="200",handler="api",method="get"} 2
ipfs_http_requests_total{code="200",handler="api",method="post"} 39
ipfs_http_requests_total{code="200",handler="gateway",method="get"} 1
ipfs_http_requests_total{code="404",handler="api",method="get"} 1
# HELP ipfs_http_response_size_bytes The HTTP response sizes in bytes.
# TYPE ipfs_http_response_size_bytes summary
ipfs_http_response_size_bytes{handler="api",quantile="0.5"} 94
ipfs_http_response_size_bytes{handler="api",quantile="0.9"} 26501
ipfs_http_response_size_bytes{handler="api",quantile="0.99"} 75009
ipfs_http_response_size_bytes_sum{handler="api"} 392770
ipfs_http_response_size_bytes_count{handler="api"} 42
ipfs_http_response_size_bytes{handler="gateway",quantile="0.5"} 19
ipfs_http_response_size_bytes{handler="gateway",quantile="0.9"} 19
ipfs_http_response_size_bytes{handler="gateway",quantile="0.99"} 19
ipfs_http_response_size_bytes_sum{handler="gateway"} 19
ipfs_http_response_size_bytes_count{handler="gateway"} 1
# HELP ipfs_http_unixfs_get_latency_seconds The time till the first block is received when 'getting' a file from the gateway.
# TYPE ipfs_http_unixfs_get_latency_seconds summary
ipfs_http_unixfs_get_latency_seconds_sum{gateway="ipfs"} 0.0001375
ipfs_http_unixfs_get_latency_seconds_count{gateway="ipfs"} 1
# HELP ipfs_info IPFS version information.
# TYPE ipfs_info gauge
ipfs_info{commit="",version="0.12.0"} 1
# HELP ipfs_p2p_peers_total Number of connected peers
# TYPE ipfs_p2p_peers_total gauge
ipfs_p2p_peers_total{transport="/ip4/tcp"} 85
ipfs_p2p_peers_total{transport="/ip4/udp/quic"} 227
# HELP leveldb_datastore_batchcommit_errors_total Number of errored Batch.Commit calls
# TYPE leveldb_datastore_batchcommit_errors_total counter
leveldb_datastore_batchcommit_errors_total 0
# HELP leveldb_datastore_batchcommit_latency_seconds Latency distribution of Batch.Commit calls
# TYPE leveldb_datastore_batchcommit_latency_seconds histogram
leveldb_datastore_batchcommit_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_batchcommit_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_batchcommit_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_batchcommit_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_batchcommit_latency_seconds_bucket{le="1"} 0
leveldb_datastore_batchcommit_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_batchcommit_latency_seconds_sum 0
leveldb_datastore_batchcommit_latency_seconds_count 0
# HELP leveldb_datastore_batchcommit_total Total number of Batch.Commit calls
# TYPE leveldb_datastore_batchcommit_total counter
leveldb_datastore_batchcommit_total 0
# HELP leveldb_datastore_batchdelete_errors_total Number of errored Batch.Delete calls
# TYPE leveldb_datastore_batchdelete_errors_total counter
leveldb_datastore_batchdelete_errors_total 0
# HELP leveldb_datastore_batchdelete_latency_seconds Latency distribution of Batch.Delete calls
# TYPE leveldb_datastore_batchdelete_latency_seconds histogram
leveldb_datastore_batchdelete_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_batchdelete_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_batchdelete_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_batchdelete_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_batchdelete_latency_seconds_bucket{le="1"} 0
leveldb_datastore_batchdelete_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_batchdelete_latency_seconds_sum 0
leveldb_datastore_batchdelete_latency_seconds_count 0
# HELP leveldb_datastore_batchdelete_total Total number of Batch.Delete calls
# TYPE leveldb_datastore_batchdelete_total counter
leveldb_datastore_batchdelete_total 0
# HELP leveldb_datastore_batchput_errors_total Number of errored Batch.Put calls
# TYPE leveldb_datastore_batchput_errors_total counter
leveldb_datastore_batchput_errors_total 0
# HELP leveldb_datastore_batchput_latency_seconds Latency distribution of Batch.Put calls
# TYPE leveldb_datastore_batchput_latency_seconds histogram
leveldb_datastore_batchput_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_batchput_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_batchput_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_batchput_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_batchput_latency_seconds_bucket{le="1"} 0
leveldb_datastore_batchput_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_batchput_latency_seconds_sum 0
leveldb_datastore_batchput_latency_seconds_count 0
# HELP leveldb_datastore_batchput_size_bytes Size distribution of byte slices put into batches
# TYPE leveldb_datastore_batchput_size_bytes histogram
leveldb_datastore_batchput_size_bytes_bucket{le="64"} 0
leveldb_datastore_batchput_size_bytes_bucket{le="4096"} 0
leveldb_datastore_batchput_size_bytes_bucket{le="262144"} 0
leveldb_datastore_batchput_size_bytes_bucket{le="1.6777216e+07"} 0
leveldb_datastore_batchput_size_bytes_bucket{le="+Inf"} 0
leveldb_datastore_batchput_size_bytes_sum 0
leveldb_datastore_batchput_size_bytes_count 0
# HELP leveldb_datastore_batchput_total Total number of Batch.Put calls
# TYPE leveldb_datastore_batchput_total counter
leveldb_datastore_batchput_total 0
# HELP leveldb_datastore_check_errors_total Number of errored Datastore.Check calls
# TYPE leveldb_datastore_check_errors_total counter
leveldb_datastore_check_errors_total 0
# HELP leveldb_datastore_check_latency_seconds Latency distribution of Datastore.Check calls
# TYPE leveldb_datastore_check_latency_seconds histogram
leveldb_datastore_check_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_check_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_check_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_check_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_check_latency_seconds_bucket{le="1"} 0
leveldb_datastore_check_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_check_latency_seconds_sum 0
leveldb_datastore_check_latency_seconds_count 0
# HELP leveldb_datastore_check_total Total number of Datastore.Check calls
# TYPE leveldb_datastore_check_total counter
leveldb_datastore_check_total 0
# HELP leveldb_datastore_delete_errors_total Number of errored Datastore.Delete calls
# TYPE leveldb_datastore_delete_errors_total counter
leveldb_datastore_delete_errors_total 0
# HELP leveldb_datastore_delete_latency_seconds Latency distribution of Datastore.Delete calls
# TYPE leveldb_datastore_delete_latency_seconds histogram
leveldb_datastore_delete_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_delete_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_delete_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_delete_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_delete_latency_seconds_bucket{le="1"} 0
leveldb_datastore_delete_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_delete_latency_seconds_sum 0
leveldb_datastore_delete_latency_seconds_count 0
# HELP leveldb_datastore_delete_total Total number of Datastore.Delete calls
# TYPE leveldb_datastore_delete_total counter
leveldb_datastore_delete_total 0
# HELP leveldb_datastore_du_errors_total Number of errored Datastore.DiskUsage calls
# TYPE leveldb_datastore_du_errors_total counter
leveldb_datastore_du_errors_total 0
# HELP leveldb_datastore_du_latency_seconds Latency distribution of Datastore.DiskUsage calls
# TYPE leveldb_datastore_du_latency_seconds histogram
leveldb_datastore_du_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_du_latency_seconds_bucket{le="0.001"} 2
leveldb_datastore_du_latency_seconds_bucket{le="0.01"} 2
leveldb_datastore_du_latency_seconds_bucket{le="0.1"} 2
leveldb_datastore_du_latency_seconds_bucket{le="1"} 2
leveldb_datastore_du_latency_seconds_bucket{le="+Inf"} 2
leveldb_datastore_du_latency_seconds_sum 0.000287416
leveldb_datastore_du_latency_seconds_count 2
# HELP leveldb_datastore_du_total Total number of Datastore.DiskUsage calls
# TYPE leveldb_datastore_du_total counter
leveldb_datastore_du_total 2
# HELP leveldb_datastore_gc_errors_total Number of errored Datastore.CollectGarbage calls
# TYPE leveldb_datastore_gc_errors_total counter
leveldb_datastore_gc_errors_total 0
# HELP leveldb_datastore_gc_latency_seconds Latency distribution of Datastore.CollectGarbage calls
# TYPE leveldb_datastore_gc_latency_seconds histogram
leveldb_datastore_gc_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_gc_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_gc_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_gc_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_gc_latency_seconds_bucket{le="1"} 0
leveldb_datastore_gc_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_gc_latency_seconds_sum 0
leveldb_datastore_gc_latency_seconds_count 0
# HELP leveldb_datastore_gc_total Total number of Datastore.CollectGarbage calls
# TYPE leveldb_datastore_gc_total counter
leveldb_datastore_gc_total 0
# HELP leveldb_datastore_get_errors_total Number of errored Datastore.Get calls
# TYPE leveldb_datastore_get_errors_total counter
leveldb_datastore_get_errors_total 0
# HELP leveldb_datastore_get_latency_seconds Latency distribution of Datastore.Get calls
# TYPE leveldb_datastore_get_latency_seconds histogram
leveldb_datastore_get_latency_seconds_bucket{le="0.0001"} 2
leveldb_datastore_get_latency_seconds_bucket{le="0.001"} 3
leveldb_datastore_get_latency_seconds_bucket{le="0.01"} 5
leveldb_datastore_get_latency_seconds_bucket{le="0.1"} 5
leveldb_datastore_get_latency_seconds_bucket{le="1"} 5
leveldb_datastore_get_latency_seconds_bucket{le="+Inf"} 5
leveldb_datastore_get_latency_seconds_sum 0.00635
leveldb_datastore_get_latency_seconds_count 5
# HELP leveldb_datastore_get_size_bytes Size distribution of retrieved byte slices
# TYPE leveldb_datastore_get_size_bytes histogram
leveldb_datastore_get_size_bytes_bucket{le="64"} 2
leveldb_datastore_get_size_bytes_bucket{le="4096"} 5
leveldb_datastore_get_size_bytes_bucket{le="262144"} 5
leveldb_datastore_get_size_bytes_bucket{le="1.6777216e+07"} 5
leveldb_datastore_get_size_bytes_bucket{le="+Inf"} 5
leveldb_datastore_get_size_bytes_sum 1169
leveldb_datastore_get_size_bytes_count 5
# HELP leveldb_datastore_get_total Total number of Datastore.Get calls
# TYPE leveldb_datastore_get_total counter
leveldb_datastore_get_total 5
# HELP leveldb_datastore_getsize_errors_total Number of errored Datastore.GetSize calls
# TYPE leveldb_datastore_getsize_errors_total counter
leveldb_datastore_getsize_errors_total 0
# HELP leveldb_datastore_getsize_latency_seconds Latency distribution of Datastore.GetSize calls
# TYPE leveldb_datastore_getsize_latency_seconds histogram
leveldb_datastore_getsize_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_getsize_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_getsize_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_getsize_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_getsize_latency_seconds_bucket{le="1"} 0
leveldb_datastore_getsize_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_getsize_latency_seconds_sum 0
leveldb_datastore_getsize_latency_seconds_count 0
# HELP leveldb_datastore_getsize_total Total number of Datastore.GetSize calls
# TYPE leveldb_datastore_getsize_total counter
leveldb_datastore_getsize_total 0
# HELP leveldb_datastore_has_errors_total Number of errored Datastore.Has calls
# TYPE leveldb_datastore_has_errors_total counter
leveldb_datastore_has_errors_total 0
# HELP leveldb_datastore_has_latency_seconds Latency distribution of Datastore.Has calls
# TYPE leveldb_datastore_has_latency_seconds histogram
leveldb_datastore_has_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_has_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_has_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_has_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_has_latency_seconds_bucket{le="1"} 0
leveldb_datastore_has_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_has_latency_seconds_sum 0
leveldb_datastore_has_latency_seconds_count 0
# HELP leveldb_datastore_has_total Total number of Datastore.Has calls
# TYPE leveldb_datastore_has_total counter
leveldb_datastore_has_total 0
# HELP leveldb_datastore_put_errors_total Number of errored Datastore.Put calls
# TYPE leveldb_datastore_put_errors_total counter
leveldb_datastore_put_errors_total 0
# HELP leveldb_datastore_put_latency_seconds Latency distribution of Datastore.Put calls
# TYPE leveldb_datastore_put_latency_seconds histogram
leveldb_datastore_put_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_put_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_put_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_put_latency_seconds_bucket{le="0.1"} 2
leveldb_datastore_put_latency_seconds_bucket{le="1"} 2
leveldb_datastore_put_latency_seconds_bucket{le="+Inf"} 2
leveldb_datastore_put_latency_seconds_sum 0.045922875
leveldb_datastore_put_latency_seconds_count 2
# HELP leveldb_datastore_put_size_bytes Size distribution of stored byte slices
# TYPE leveldb_datastore_put_size_bytes histogram
leveldb_datastore_put_size_bytes_bucket{le="64"} 0
leveldb_datastore_put_size_bytes_bucket{le="4096"} 2
leveldb_datastore_put_size_bytes_bucket{le="262144"} 2
leveldb_datastore_put_size_bytes_bucket{le="1.6777216e+07"} 2
leveldb_datastore_put_size_bytes_bucket{le="+Inf"} 2
leveldb_datastore_put_size_bytes_sum 782
leveldb_datastore_put_size_bytes_count 2
# HELP leveldb_datastore_put_total Total number of Datastore.Put calls
# TYPE leveldb_datastore_put_total counter
leveldb_datastore_put_total 2
# HELP leveldb_datastore_query_errors_total Number of errored Datastore.Query calls
# TYPE leveldb_datastore_query_errors_total counter
leveldb_datastore_query_errors_total 0
# HELP leveldb_datastore_query_latency_seconds Latency distribution of Datastore.Query calls
# TYPE leveldb_datastore_query_latency_seconds histogram
leveldb_datastore_query_latency_seconds_bucket{le="0.0001"} 1
leveldb_datastore_query_latency_seconds_bucket{le="0.001"} 1
leveldb_datastore_query_latency_seconds_bucket{le="0.01"} 1
leveldb_datastore_query_latency_seconds_bucket{le="0.1"} 1
leveldb_datastore_query_latency_seconds_bucket{le="1"} 1
leveldb_datastore_query_latency_seconds_bucket{le="+Inf"} 1
leveldb_datastore_query_latency_seconds_sum 4.9542e-05
leveldb_datastore_query_latency_seconds_count 1
# HELP leveldb_datastore_query_total Total number of Datastore.Query calls
# TYPE leveldb_datastore_query_total counter
leveldb_datastore_query_total 1
# HELP leveldb_datastore_scrub_errors_total Number of errored Datastore.Scrub calls
# TYPE leveldb_datastore_scrub_errors_total counter
leveldb_datastore_scrub_errors_total 0
# HELP leveldb_datastore_scrub_latency_seconds Latency distribution of Datastore.Scrub calls
# TYPE leveldb_datastore_scrub_latency_seconds histogram
leveldb_datastore_scrub_latency_seconds_bucket{le="0.0001"} 0
leveldb_datastore_scrub_latency_seconds_bucket{le="0.001"} 0
leveldb_datastore_scrub_latency_seconds_bucket{le="0.01"} 0
leveldb_datastore_scrub_latency_seconds_bucket{le="0.1"} 0
leveldb_datastore_scrub_latency_seconds_bucket{le="1"} 0
leveldb_datastore_scrub_latency_seconds_bucket{le="+Inf"} 0
leveldb_datastore_scrub_latency_seconds_sum 0
leveldb_datastore_scrub_latency_seconds_count 0
# HELP leveldb_datastore_scrub_total Total number of Datastore.Scrub calls
# TYPE leveldb_datastore_scrub_total counter
leveldb_datastore_scrub_total 0
# HELP leveldb_datastore_sync_errors_total Number of errored Datastore.Sync calls
# TYPE leveldb_datastore_sync_errors_total counter
leveldb_datastore_sync_errors_total 0
# HELP leveldb_datastore_sync_latency_seconds Latency distribution of Datastore.Sync calls
# TYPE leveldb_datastore_sync_latency_seconds histogram
leveldb_datastore_sync_latency_seconds_bucket{le="0.0001"} 1
leveldb_datastore_sync_latency_seconds_bucket{le="0.001"} 1
leveldb_datastore_sync_latency_seconds_bucket{le="0.01"} 1
leveldb_datastore_sync_latency_seconds_bucket{le="0.1"} 1
leveldb_datastore_sync_latency_seconds_bucket{le="1"} 1
leveldb_datastore_sync_latency_seconds_bucket{le="+Inf"} 1
leveldb_datastore_sync_latency_seconds_sum 1.292e-06
leveldb_datastore_sync_latency_seconds_count 1
# HELP leveldb_datastore_sync_total Total number of Datastore.Sync calls
# TYPE leveldb_datastore_sync_total counter
leveldb_datastore_sync_total 1
# HELP quic_connection_duration Connection Duration
# TYPE quic_connection_duration histogram
quic_connection_duration_bucket{le="1"} 0
quic_connection_duration_bucket{le="1.5"} 0
quic_connection_duration_bucket{le="2.25"} 3
quic_connection_duration_bucket{le="3.375"} 3
quic_connection_duration_bucket{le="5.0625"} 3
quic_connection_duration_bucket{le="7.59375"} 3
quic_connection_duration_bucket{le="11.390625"} 3
quic_connection_duration_bucket{le="17.0859375"} 18
quic_connection_duration_bucket{le="25.62890625"} 38
quic_connection_duration_bucket{le="38.443359375"} 227
quic_connection_duration_bucket{le="57.6650390625"} 393
quic_connection_duration_bucket{le="86.49755859375"} 468
quic_connection_duration_bucket{le="129.746337890625"} 500
quic_connection_duration_bucket{le="194.6195068359375"} 500
quic_connection_duration_bucket{le="291.92926025390625"} 500
quic_connection_duration_bucket{le="437.8938903808594"} 500
quic_connection_duration_bucket{le="656.8408355712891"} 500
quic_connection_duration_bucket{le="985.2612533569336"} 500
quic_connection_duration_bucket{le="1477.8918800354004"} 500
quic_connection_duration_bucket{le="2216.8378200531006"} 500
quic_connection_duration_bucket{le="3325.256730079651"} 500
quic_connection_duration_bucket{le="4987.885095119476"} 500
quic_connection_duration_bucket{le="7481.8276426792145"} 500
quic_connection_duration_bucket{le="11222.741464018822"} 500
quic_connection_duration_bucket{le="16834.112196028233"} 500
quic_connection_duration_bucket{le="25251.16829404235"} 500
quic_connection_duration_bucket{le="37876.75244106352"} 500
quic_connection_duration_bucket{le="56815.128661595285"} 500
quic_connection_duration_bucket{le="85222.69299239293"} 500
quic_connection_duration_bucket{le="127834.03948858939"} 500
quic_connection_duration_bucket{le="191751.0592328841"} 500
quic_connection_duration_bucket{le="287626.58884932613"} 500
quic_connection_duration_bucket{le="431439.8832739892"} 500
quic_connection_duration_bucket{le="647159.8249109838"} 500
quic_connection_duration_bucket{le="970739.7373664756"} 500
quic_connection_duration_bucket{le="1.4561096060497134e+06"} 500
quic_connection_duration_bucket{le="2.18416440907457e+06"} 500
quic_connection_duration_bucket{le="3.276246613611855e+06"} 500
quic_connection_duration_bucket{le="4.914369920417783e+06"} 500
quic_connection_duration_bucket{le="7.371554880626675e+06"} 500
quic_connection_duration_bucket{le="+Inf"} 500
quic_connection_duration_sum 22701.034386958
quic_connection_duration_count 500
# HELP quic_connection_errors_total QUIC connection errors
# TYPE quic_connection_errors_total counter
quic_connection_errors_total{error_code="APPLICATION_ERROR",side="remote"} 5
quic_connection_errors_total{error_code="CRYPTO_ERROR (0x12a)",side="local"} 3
quic_connection_errors_total{error_code="CRYPTO_ERROR (0x12a)",side="remote"} 71
quic_connection_errors_total{error_code="INTERNAL_ERROR",side="local"} 183
quic_connection_errors_total{error_code="idle_timeout",side="local"} 68
quic_connection_errors_total{error_code="stateless_reset",side="remote"} 1
quic_connection_errors_total{error_code="version_negotiation",side="local"} 1
# HELP quic_connections_closed_total closed QUIC connection
# TYPE quic_connections_closed_total counter
quic_connections_closed_total{direction="outgoing"} 79
# HELP quic_connections_new_total new QUIC connection
# TYPE quic_connections_new_total counter
quic_connections_new_total{direction="incoming",handshake_successful="false"} 63
quic_connections_new_total{direction="outgoing",handshake_successful="false"} 608
quic_connections_new_total{direction="outgoing",handshake_successful="true"} 305
# HELP quic_packets_buffered_total Buffered packets
# TYPE quic_packets_buffered_total counter
quic_packets_buffered_total{packet_type="1-RTT"} 1
# HELP quic_packets_dropped_total Dropped packets
# TYPE quic_packets_dropped_total counter
quic_packets_dropped_total{packet_type="Handshake",reason="key_unavailable"} 3
quic_packets_dropped_total{packet_type="Initial",reason="key_unavailable"} 186
# HELP quic_packets_lost_total QUIC lost received
# TYPE quic_packets_lost_total counter
quic_packets_lost_total{encryption_level="1-RTT",reason="time_threshold"} 2
# HELP quic_packets_rcvd_total QUIC packets received
# TYPE quic_packets_rcvd_total counter
quic_packets_rcvd_total{encryption_level="1-RTT"} 9436
quic_packets_rcvd_total{encryption_level="Handshake"} 792
quic_packets_rcvd_total{encryption_level="Initial"} 447
quic_packets_rcvd_total{encryption_level="Version Negotiation"} 1
# HELP quic_packets_sent_total QUIC packets sent
# TYPE quic_packets_sent_total counter
quic_packets_sent_total{encryption_level="1-RTT"} 7818
quic_packets_sent_total{encryption_level="Handshake"} 697
quic_packets_sent_total{encryption_level="Initial"} 3180
# HELP quic_smoothed_rtt Smoothed RTT
# TYPE quic_smoothed_rtt histogram
quic_smoothed_rtt_bucket{le="0.001"} 0
quic_smoothed_rtt_bucket{le="0.00125"} 0
quic_smoothed_rtt_bucket{le="0.0015625"} 0
quic_smoothed_rtt_bucket{le="0.001953125"} 0
quic_smoothed_rtt_bucket{le="0.00244140625"} 0
quic_smoothed_rtt_bucket{le="0.0030517578125"} 0
quic_smoothed_rtt_bucket{le="0.003814697265625"} 0
quic_smoothed_rtt_bucket{le="0.00476837158203125"} 0
quic_smoothed_rtt_bucket{le="0.0059604644775390625"} 0
quic_smoothed_rtt_bucket{le="0.007450580596923828"} 0
quic_smoothed_rtt_bucket{le="0.009313225746154785"} 0
quic_smoothed_rtt_bucket{le="0.011641532182693481"} 2
quic_smoothed_rtt_bucket{le="0.014551915228366852"} 7
quic_smoothed_rtt_bucket{le="0.018189894035458565"} 9
quic_smoothed_rtt_bucket{le="0.022737367544323206"} 9
quic_smoothed_rtt_bucket{le="0.028421709430404007"} 18
quic_smoothed_rtt_bucket{le="0.03552713678800501"} 26
quic_smoothed_rtt_bucket{le="0.04440892098500626"} 32
quic_smoothed_rtt_bucket{le="0.05551115123125783"} 38
quic_smoothed_rtt_bucket{le="0.06938893903907228"} 56
quic_smoothed_rtt_bucket{le="0.08673617379884035"} 150
quic_smoothed_rtt_bucket{le="0.10842021724855044"} 193
quic_smoothed_rtt_bucket{le="0.13552527156068805"} 208
quic_smoothed_rtt_bucket{le="0.16940658945086007"} 288
quic_smoothed_rtt_bucket{le="0.21175823681357508"} 392
quic_smoothed_rtt_bucket{le="0.26469779601696886"} 456
quic_smoothed_rtt_bucket{le="0.33087224502121104"} 485
quic_smoothed_rtt_bucket{le="0.41359030627651383"} 491
quic_smoothed_rtt_bucket{le="0.5169878828456422"} 493
quic_smoothed_rtt_bucket{le="0.6462348535570528"} 495
quic_smoothed_rtt_bucket{le="0.807793566946316"} 495
quic_smoothed_rtt_bucket{le="1.009741958682895"} 495
quic_smoothed_rtt_bucket{le="1.2621774483536188"} 495
quic_smoothed_rtt_bucket{le="1.5777218104420234"} 495
quic_smoothed_rtt_bucket{le="1.9721522630525292"} 495
quic_smoothed_rtt_bucket{le="2.4651903288156616"} 495
quic_smoothed_rtt_bucket{le="3.081487911019577"} 495
quic_smoothed_rtt_bucket{le="3.8518598887744715"} 495
quic_smoothed_rtt_bucket{le="4.8148248609680895"} 495
quic_smoothed_rtt_bucket{le="6.018531076210111"} 495
quic_smoothed_rtt_bucket{le="+Inf"} 495
quic_smoothed_rtt_sum 75.31521499999998
quic_smoothed_rtt_count 495
# HELP quic_transferred_bytes QUIC bytes transferred
# TYPE quic_transferred_bytes counter
quic_transferred_bytes{direction="rcvd"} 3.337995e+06
quic_transferred_bytes{direction="sent"} 6.206e+06
# HELP tcp_connection_duration TCP Connection Duration
# TYPE tcp_connection_duration histogram
tcp_connection_duration_bucket{le="1"} 12
tcp_connection_duration_bucket{le="1.5"} 22
tcp_connection_duration_bucket{le="2.25"} 32
tcp_connection_duration_bucket{le="3.375"} 46
tcp_connection_duration_bucket{le="5.0625"} 53
tcp_connection_duration_bucket{le="7.59375"} 79
tcp_connection_duration_bucket{le="11.390625"} 122
tcp_connection_duration_bucket{le="17.0859375"} 179
tcp_connection_duration_bucket{le="25.62890625"} 276
tcp_connection_duration_bucket{le="38.443359375"} 405
tcp_connection_duration_bucket{le="57.6650390625"} 484
tcp_connection_duration_bucket{le="86.49755859375"} 539
tcp_connection_duration_bucket{le="129.746337890625"} 557
tcp_connection_duration_bucket{le="194.6195068359375"} 557
tcp_connection_duration_bucket{le="291.92926025390625"} 557
tcp_connection_duration_bucket{le="437.8938903808594"} 557
tcp_connection_duration_bucket{le="656.8408355712891"} 557
tcp_connection_duration_bucket{le="985.2612533569336"} 557
tcp_connection_duration_bucket{le="1477.8918800354004"} 557
tcp_connection_duration_bucket{le="2216.8378200531006"} 557
tcp_connection_duration_bucket{le="3325.256730079651"} 557
tcp_connection_duration_bucket{le="4987.885095119476"} 557
tcp_connection_duration_bucket{le="7481.8276426792145"} 557
tcp_connection_duration_bucket{le="11222.741464018822"} 557
tcp_connection_duration_bucket{le="16834.112196028233"} 557
tcp_connection_duration_bucket{le="25251.16829404235"} 557
tcp_connection_duration_bucket{le="37876.75244106352"} 557
tcp_connection_duration_bucket{le="56815.128661595285"} 557
tcp_connection_duration_bucket{le="85222.69299239293"} 557
tcp_connection_duration_bucket{le="127834.03948858939"} 557
tcp_connection_duration_bucket{le="191751.0592328841"} 557
tcp_connection_duration_bucket{le="287626.58884932613"} 557
tcp_connection_duration_bucket{le="431439.8832739892"} 557
tcp_connection_duration_bucket{le="647159.8249109838"} 557
tcp_connection_duration_bucket{le="970739.7373664756"} 557
tcp_connection_duration_bucket{le="1.4561096060497134e+06"} 557
tcp_connection_duration_bucket{le="2.18416440907457e+06"} 557
tcp_connection_duration_bucket{le="3.276246613611855e+06"} 557
tcp_connection_duration_bucket{le="4.914369920417783e+06"} 557
tcp_connection_duration_bucket{le="7.371554880626675e+06"} 557
tcp_connection_duration_bucket{le="+Inf"} 557
tcp_connection_duration_sum 16666.24537123899
tcp_connection_duration_count 557
# HELP tcp_connections_closed_total TCP connections closed
# TYPE tcp_connections_closed_total counter
tcp_connections_closed_total{direction="incoming"} 134
tcp_connections_closed_total{direction="outgoing"} 634
# HELP tcp_connections_new_total TCP new connections
# TYPE tcp_connections_new_total counter
tcp_connections_new_total{direction="incoming"} 69
tcp_connections_new_total{direction="outgoing"} 419
# HELP tcp_rcvd_bytes TCP bytes received
# TYPE tcp_rcvd_bytes counter
tcp_rcvd_bytes 624909
# HELP tcp_rcvd_segments_total TCP segments received
# TYPE tcp_rcvd_segments_total counter
tcp_rcvd_segments_total 1565
# HELP tcp_rtt TCP round trip time
# TYPE tcp_rtt histogram
tcp_rtt_bucket{le="0.001"} 27
tcp_rtt_bucket{le="0.00125"} 27
tcp_rtt_bucket{le="0.0015625"} 27
tcp_rtt_bucket{le="0.001953125"} 27
tcp_rtt_bucket{le="0.00244140625"} 27
tcp_rtt_bucket{le="0.0030517578125"} 27
tcp_rtt_bucket{le="0.003814697265625"} 27
tcp_rtt_bucket{le="0.00476837158203125"} 27
tcp_rtt_bucket{le="0.0059604644775390625"} 27
tcp_rtt_bucket{le="0.007450580596923828"} 27
tcp_rtt_bucket{le="0.009313225746154785"} 27
tcp_rtt_bucket{le="0.011641532182693481"} 27
tcp_rtt_bucket{le="0.014551915228366852"} 27
tcp_rtt_bucket{le="0.018189894035458565"} 34
tcp_rtt_bucket{le="0.022737367544323206"} 38
tcp_rtt_bucket{le="0.028421709430404007"} 40
tcp_rtt_bucket{le="0.03552713678800501"} 40
tcp_rtt_bucket{le="0.04440892098500626"} 45
tcp_rtt_bucket{le="0.05551115123125783"} 55
tcp_rtt_bucket{le="0.06938893903907228"} 56
tcp_rtt_bucket{le="0.08673617379884035"} 101
tcp_rtt_bucket{le="0.10842021724855044"} 116
tcp_rtt_bucket{le="0.13552527156068805"} 123
tcp_rtt_bucket{le="0.16940658945086007"} 253
tcp_rtt_bucket{le="0.21175823681357508"} 407
tcp_rtt_bucket{le="0.26469779601696886"} 448
tcp_rtt_bucket{le="0.33087224502121104"} 486
tcp_rtt_bucket{le="0.41359030627651383"} 511
tcp_rtt_bucket{le="0.5169878828456422"} 514
tcp_rtt_bucket{le="0.6462348535570528"} 523
tcp_rtt_bucket{le="0.807793566946316"} 526
tcp_rtt_bucket{le="1.009741958682895"} 526
tcp_rtt_bucket{le="1.2621774483536188"} 532
tcp_rtt_bucket{le="1.5777218104420234"} 533
tcp_rtt_bucket{le="1.9721522630525292"} 533
tcp_rtt_bucket{le="2.4651903288156616"} 533
tcp_rtt_bucket{le="3.081487911019577"} 535
tcp_rtt_bucket{le="3.8518598887744715"} 538
tcp_rtt_bucket{le="4.8148248609680895"} 538
tcp_rtt_bucket{le="6.018531076210111"} 539
tcp_rtt_bucket{le="+Inf"} 557
tcp_rtt_sum 457.51399999999967
tcp_rtt_count 557
# HELP tcp_sent_bytes TCP bytes sent
# TYPE tcp_sent_bytes counter
tcp_sent_bytes 138689
# HELP tcp_sent_segments_total TCP segments sent
# TYPE tcp_sent_segments_total counter
tcp_sent_segments_total 1341

Is there another method of consuming this data?

@lidel
Copy link
Member

lidel commented Apr 21, 2022

My quick take is:

  • we want to provide ipfs-webui (go-ipfs, ipfs-desktop) users with a quick view for exploring metrics without the need for setting up Expression Browser or Grafana.
  • rationale: text is fine for basic count, but having useful visualization of histogram metrics makes it way easier to answer questions like:
    • is my gateway slow to respond, or the problem reported by the user is just an outlier? if so, how fast is 95th percent of responses?
    • what is the most popular gateway response type? which request type takes the most of time?
    • how long most of quic/tcp connections last?
    • is there a visible degradation in the duration of my datastores' reads/writes?
    • etc

So in practice, we can't hardcode or assume too much:

  • Figure out how/where to add "Diagnostic" screen to ipfs-webui
    • Make it, so it can have multiple tabs (we want metrics, but we will also add logs, and other diagnostic tooling there over time, so good to create foundation for that from the start)
  • "Metrics" tab should be a generic solution
    • Show it only when debug/metrics/prometheus endpoint is available, and provide UI for exploring metrics listed there.
    • Dynamic discovery: assume new metrics can be added over time, and old metrics can be removed or deprecated, so we can't expect anything to stay the same.
    • Good news is that this wall of text is becoming a standard(?):
      1. https://prometheus.io/docs/instrumenting/exposition_formats/#openmetrics-text-format
      2. https://github.com/OpenObservability/OpenMetrics
      3. Given that each metric has standard TYPE like counter and histogram and unique name and description defined in HELP lines, we should be able to generate UI for each (if there is no JS tooling for this format already -- needs research)

@olizilla
Copy link
Member

olizilla commented Apr 22, 2022

On my way to work this morning I started day-dreaming about how nice it would be to get back to working on developer tools for IPFS & co. A dangerous move as I was riding my bike at the time, but lo! What serendipitous timing! I'd love to pitch in on this!

That wall of text is the only format for getting that particular list of data points. It's intended for consumption by Prometheus, the timeseries db, but it's well standardised and much tooling exists for it. The numbers are all point-in-time measurements, intended to be scraped periodically, so it's not such a bad fit for a local dev tool that is already in the habit of polling the api for info.

There are some useful metrics in there that we use all the time now, to check if a node is working well, things like bitswap queue length, but in general much of the metrics are super specific, and only really of use to a developer who is making specific changes to a specific subsystem, so we should build up a short list of metrics that are worth putting the spotlight on, and then give experts a way to get to the kitchen sink.

@olizilla
Copy link
Member

background concern:

  • the data points available at debug/metrics/prometheus have grown organically over time... I'm not sure they are all useful or form a coherent overview of what is going on inside a go-ipfs node... my assumption is that the list represents a set of things that needed some performance tuning at some point.

What questions are we trying to answer with this view?
What problems do we expect this view to solve, or assist with solving?

This is a very good place to start... I don't have the answers, but it will involve talking to users and seeing what problems they are hitting.

In general, there is a need for high quality dev tooling for ipfs / libp2p / ipld ...when i proposed parsing info from debug/metrics/prometheus, I wasn't familiar with prometheus and was groping around in the dark for anything that might be interesting.

Now I think we can use it to prototype something and get a feel for it, but we should be careful to not end up implementing an incidental subset of prometheus in js. Ideally we also use this work to improve the core ipfs diagnostic apis as well. I think as @SgtPooki is pointing to, parsing info from prometheus metrics for the web UI is not ideal.

@SgtPooki SgtPooki changed the title Feature: Create a diagnostics & metrics view Feature: Create a metrics view Apr 25, 2022
@SgtPooki SgtPooki added this to the New Metrics Screen milestone May 9, 2022
@SgtPooki SgtPooki added the area/screen/metrics Issues relating to the metrics screen label May 9, 2022
@SgtPooki SgtPooki added need/maintainers-input Needs input from the current maintainer(s) need/community-input Needs input from the wider community labels May 24, 2022
@lidel
Copy link
Member

lidel commented Jun 20, 2022

Highly relevant work is happening in:

When we have these go-libp2p improvements released and bubbled up to kubo (go-ipfs), ipfs-webui will be able to benefit from metrics and work done by Marco around fleshing out important metrics and ways to visualize them.

@SgtPooki
Copy link
Member

There is a large amount of metrics things at https://github.com/protocol/metrics

@RamilMus
Copy link

Hello!
Looked through your discussion and thought it is very relevant. Please tell me if I need to open a separate issue.
Can you please help with quic_connection_errors_total metric. When Prometheus scrapes the path http://localhost:5001/debug/metrics/prometheus it gets thousands of lines with this error metric which are not combined, as a result prometheus marks this exporter as down due to time out. Is there a way to disable them?

@SgtPooki SgtPooki removed their assignment Jul 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/screen/metrics Issues relating to the metrics screen effort/days Estimated to take multiple days, but less than a week exp/intermediate Prior experience is likely helpful kind/discussion Topical discussion; usually not changes to codebase kind/enhancement A net-new feature or improvement to an existing feature need/analysis Needs further analysis before proceeding need/community-input Needs input from the wider community need/maintainers-input Needs input from the current maintainer(s) P3 Low: Not priority right now status/deferred Conscious decision to pause or backlog topic/design-front-end Front-end implementation of UX/UI work topic/design-ux UX strategy, research, not solely visual design
Projects
No open projects
Status: Needs Prep Work / Blocked
Development

No branches or pull requests

6 participants