Skip to content

Latest commit

 

History

History
178 lines (163 loc) · 38.1 KB

metricDetails.md

File metadata and controls

178 lines (163 loc) · 38.1 KB
Raw

Details of Metrics Collected by the JVM-Profiler

  1. CPU and Memory
  2. IO
  3. Process Info
  4. Stack Trace

1. CPU and Memory

Name Description Sample Value Note
time Time series value for grafana database. 1582504748538000000
appId Application id of the job that is running. application_1576797426048_311346 If using yarn, this is the yarn application id.
bufferPools-direct-count Total number of direct byte buffers used at the recorded time. 55 Depending on your application, this can be a smaller number
bufferPools-direct-memoryUsed Total memory of the direct byte buffers used in bytes. 18950 An estimate. This value should be very similar to the total capacity for a healthy JVM.
bufferPools-direct-name Name of the buffer pool. direct Direct buffer pools are always named direct.
bufferPools-direct-totalCapacity Available capacity in bytes in the direct buffer pool. 18949 Capacity should be similar to memoryUsed for a healthy JVM.
bufferPools-mapped-count Total number of mapped byte buffers used at the recorded time. 0 In general, spark applications will not require mapped buffer pools.
bufferPools-mapped-memoryUsed Total memory of the mapped byte buffers used in bytes. 0 An estimate. This value should be very similar to the total capacity for a healthy JVM.
bufferPools-mapped-name Name of the buffer pool. mapped Mapped buffer pools are always named mapped.
bufferPools-mapped-totalCapacity Available capacity in bytes in the mapped buffer pool. 0 Capacity should be similar to memoryUsed for a healthy JVM.
epochMillis System time in milliseconds when this value was recorded. 1582504781508
gc-PSMarkSweep-collectionCount Total number of collections that have occurred for PS MarkSweep. 7 -1 if collection count is undefined for this collector.
gc-PSMarkSweep-collectionTime approximate accumulated collection elapsed time in milliseconds. 914 -1 if collection count is undefined for this collector.
gc-PSMarkSweep-name Name of the collector. PS MarkSweep Constant value.
gc-PSScavenge-collectionCount Total number of collections that have occurred for PS Scavenge. 25 -1 if collection count is undefined for this collector.
gc-PSScavenge-collectionTime approximate accumulated collection elapsed time in milliseconds. 1206 -1 if collection count is undefined for this collector.
gc-PSScavenge-name Name of the collector. PS Scavenge Constant value.
heapMemoryCommitted Amount of memory in bytes guaranteed for the JVM to use. 375914496
heapMemoryMax Max amount of memory in bytes that can be used 11453595648 Also refer to spark.memory.fraction for the fraction managed by spark
heapMemoryTotalUsed Total heap memory used in bytes 86386032 Total memory used should be less than committed and max heap memory.
host Hostname of the executor uber-bigdata-datanode2.uber.com
memoryPools-CodeCache-name Name of the code cache. Code Cache Constant value.
memoryPools-CodeCache-peakUsageCommitted Peak committed memory of this pool since the JVM was started 2555904
memoryPools-CodeCache-peakUsageMax Peak value of usageMax of this pool since the JVM was started 251658240
memoryPools-CodeCache-peakUsageUsed Peak value of memory of this pool used since the JVM was started. 582656
memoryPools-CodeCache-type The type of this memory pool. Non-heap memory Code cache is a non-heap memory. So this value will always be constant.
memoryPools-CodeCache-usageCommitted Total memory in bytes committed for this pool. 2555904
memoryPools-CodeCache-usageMax The maximum amount of memory that can be used by this pool. 251658240 -1 if undefined. Not guaranteed to be available if greater than the committed memory.
memoryPools-CodeCache-usageUsed Amount of memory used by this pool. 582656 Having some code cache memory free ensures it is usable for JIT compilation.
memoryPools-CompressedClassSpace-name Name of the compressed class space. Compressed Class Space Is constant. This space is allocated in native memory for class metadata.
memoryPools-CompressedClassSpace-peakUsageCommitted Peak committed memory of this pool since JVM was started. 1572864
memoryPools-CompressedClassSpace-peakUsageMax Peak value of usageMax of this pool since the JVM was started. 1073741824
memoryPools-CompressedClassSpace-peakUsageUsed Peak value of usage of this pool since the JVM was started. 1389800
memoryPools-CompressedClassSpace-type Type of memory of this memory pool. Non-heap memory Compressed class space is native memory but is managed by the JVM
memoryPools-CompressedClassSpace-usageCommitted Total memory in bytes committed for this pool. 1572864
memoryPools-CompressedClassSpace-usageMax The maximum amount of memory that can be used by this pool 1073741824 -1 if undefined. Not guaranteed to be available if greater than the committed memory.
memoryPools-CompressedClassSpace-usageUsed Amount of memory used by this pool. 1389800
memoryPools-Metaspace-name Name of the Metaspace. Metaspace Is constant. Metaspace is always called metaspace.
memoryPools-Metaspace-peakUsageCommitted Peak committed memory of this pool since JVM was started. 12320768
memoryPools-Metaspace-peakUsageMax Peak value of usageMax of this pool since the JVM was started. -1 -1 if undefined.
memoryPools-Metaspace-peakUsageUsed Peak value of usage of this pool since the JVM was started. 11768752
memoryPools-Metaspace-type Type of memory of this memory pool. Non-heap memory Metaspace is native memory (since JDK 8) but is managed by the JVM
memoryPools-Metaspace-usageCommitted Total memory in bytes committed for this pool. 12320768
memoryPools-Metaspace-usageMax The maximum amount of memory that can be used by this pool -1 -1 if undefined. Not guaranteed to be available if greater than the committed memory.
memoryPools-Metaspace-usageUsed Amount of memory used by this pool. 11768752 The amount of space used by loaded classes.
memoryPools-PSEdenSpace-name Name of the young generation heap space. PS Eden Space Is constant. PS Eden space is always called PS Eden Space.
memoryPools-PSEdenSpace-peakUsageCommitted Peak committed memory of this pool since JVM was started. 98041856
memoryPools-PSEdenSpace-peakUsageMax Peak value of usageMax of this pool since the JVM was started. 4262461440 -1 if undefined.
memoryPools-PSEdenSpace-peakUsageUsed Peak value of usage of this pool since the JVM was started. 92270136
memoryPools-PSEdenSpace-type Type of memory of this memory pool. Heap memory PS Eden space is a portion of the heap memory that is managed by the JVM
memoryPools-PSEdenSpace-usageCommitted Total memory in bytes committed for this pool. 98041856
memoryPools-PSEdenSpace-usageMax The maximum amount of memory that can be used by this pool 8589934592 -1 if undefined. Not guaranteed to be available if greater than the committed memory.
memoryPools-PSEdenSpace-usageUsed Amount of memory used by this pool (and by newly loaded classes). 92270136 A continuously increasing value indicates an unhealthy garbage collection.
memoryPools-PSOldGen-name Name of the old generation heap space. PS Old Gen Is constant. PS Old Gen space is always called PS Old Gen.
memoryPools-PSOldGen-peakUsageCommitted Peak committed memory of this pool since JVM was started. 261619712
memoryPools-PSOldGen-peakUsageMax Peak value of usageMax of this pool since the JVM was started. 8589934592 -1 if undefined.
memoryPools-PSOldGen-peakUsageUsed Peak value of usage of this pool since the JVM was started. 0
memoryPools-PSOldGen-type Type of memory of this memory pool. Heap memory PS Old Gen space is a portion of the heap memory that is managed by the JVM
memoryPools-PSOldGen-usageCommitted Total memory in bytes committed for this pool. 261619712
memoryPools-PSOldGen-usageMax The maximum amount of memory that can be used by this pool 8589934592 -1 if undefined. Not guaranteed to be available if greater than the committed memory.
memoryPools-PSOldGen-usageUsed Amount of memory used by this pool (by objects beyond threshold). 0 A continuously increasing value indicates an unhealthy garbage collection.
memoryPools-PSSurvivorSpace-name Name of the survivor heap space. PS Survivor Space Is constant. PS Survivor Space is always called PS Survivor Space.
memoryPools-PSSurvivorSpace-peakUsageCommitted Peak committed memory of this pool since JVM was started. 16252928
memoryPools-PSSurvivorSpace-peakUsageMax Peak value of usageMax of this pool since the JVM was started. 16252928 -1 if undefined.
memoryPools-PSSurvivorSpace-peakUsageUsed Peak value of usage of this pool since the JVM was started. 0
memoryPools-PSSurvivorSpace-type Type of memory of this memory pool. Heap memory PS Survivor Space is a portion of the heap memory that is managed by the JVM
memoryPools-PSSurvivorSpace-usageCommitted Total memory in bytes committed for this pool. 16252928
memoryPools-PSSurvivorSpace-usageMax The maximum amount of memory that can be used by this pool 16252928 -1 if undefined. Not guaranteed to be available if greater than the committed memory.
memoryPools-PSSurvivorSpace-usageUsed Amount of memory used by this pool (and by newly loaded classes). 0 A continuously increasing value indicates an unhealthy garbage collection.
name Process name (number) 12345@uber-bigdata-datanode2.uber.com Also contains host name.
nonHeapMemoryCommitted Amount of non heap memory in bytes committed for the JVM to use. 16187392 Total of committed of all non-heap memory pools.
nonHeapMemoryMax Maximum amount of memory in bytes that can be used. -1 -1 if undefined.
nonHeapMemoryTotalUsed Amount of non heap memory (metaspace, code cache) used. 13510368 Total of usage of all non-heap memory pools.
processCpuLoad The recent cpu usage for the JVM process. 4.335671366438364e-9
processCpuTime CPU time used by JVM process in nanoseconds. 2340000000
processUuid UUID of the JVM process. 48abf05b-dfd4-4069-86c6-8cdbc195bd40
processUuid_1 UUID of the JVM process. 48abf05b-dfd4-4069-86c6-8cdbc195bd40 Appears to be a duplicate metric.
role Role of this spark executor. executor Can be either executor or driver.
systemCpuLoad The recent cpu usage for the whole system. 0.03703887412577616
vmHWM Peak resident set size ("High Water Mark") in bytes for the JVM. 142876672 An indication of actual physical memory (RAM) usage in linux by the JVM.
vmPeak Peak of the virtual memory usage in bytes for the JVM. 15130554368
vmRSS JVM physical memory usage (resident set size) in bytes. 142876672
vmSize Virtual memory size for the JVM. 15130550272 This memory includes swap space, so a large value can imply high swap memory usage.

Back to Top

2. IO

Name Description Sample Value Notes
time Time series value for grafana database. 1582504713009000000
appId Application id for the job that is running. application_1576797426048_311346 If using yarn, this is the yarn application id.
epochMillis System time in milliseconds when this value was recorded. 1582504713001
host Hostname of the executor. uber-bigdata-datanode2.uber.com
name Process name (number). 12345@uber-bigdata-datanode2.uber.com Includes the hostname as well.
processUuid UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743
processUuid_1 UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743 Appears to be a duplicate metric.
role Role of this spark executor. executor Can be either executor or driver.
self-io-rchar-1 The number of characters read by this JVM process. 8192340 The "-1" is coming from the formatter for influx which was used when writting this.
self-io-read_bytes-1 Attempt to count the bytes read by this JVM process from storage. 1892352
self-io-wchar-1 The number of characters written to disk by this JVM process. 51016
self-io-write_bytes-1 Attempt to count the bytes written by JVM process to storage. 86016
stat-cpu-N CPU entry found in /proc//stat. cpu, cpu0, cpu1, cpu2 ... N is the CPU number starting at cpu-1 (i.e. cpu) which aggregates all other CPU lines.
stat-idle-N Time the N-th CPU spent in the idle task. 53679018 Time is in USER_HZ (1/100th of a second).
stat-iowait-N Time spent by N-th CPU waiting for I/O to complete. 137645010
stat-nice-N Time spent by N-th CPU in user mode with low priority (nice). 104 Low idle time with high nice time are indications of busy background tasks running.
stat-system-N Time spent by N-th CPU in system mode. 32700364 Time spent at kernel level.
stat-user-N Time spent by N-th CPU at the user (application) level 279816973

Back to Top

3. Process Info

Name Description Sample Value Notes
time Time series value for grafana database. 1582504713009000000
agentVersion A constant value defined in code. 1.0.0 Agent version will always be 1.0.0.
appId Application id for the job that is running. application_1576797426048_311346 If using yarn, this is the yarn application id.
cmdline Contains JVM command executed and input arguments and classpath. /usr/jdk64/jdk1.8.0_112/bin/java -javaagent...
epochMillis System time in milliseconds when this value was recorded. 1582504713001
host Hostname of the executor. uber-bigdata-datanode2.uber.com
jvmClassPath Contains JVM classpath entries. --user-class-path .../jvm-profiler-1.0.0.jar This entry is emtpy if cmdLine is not empty. This is to avoid duplicate information.
jvmInputArguments Contains JVM input arguments. metricInterval=5000,sampleInterval=5000 ... This entry is emtpy if cmdLine is not empty. This is to avoid duplicate information.
name Process name (number). 12345@uber-bigdata-datanode2.uber.com Includes the hostname as well.
processUuid UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743
processUuid_1 UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743 Appears to be a duplicate metric.
role Role of this spark executor. executor Can be either executor or driver.
xmxBytes Maximum JVM heap size in bytes. 12884901888

Back to Top

4. Stack Trace

Name Description Sample Value Notes
time Time series value for grafana database. 1582504713009000000
appId Application id for the job that is running. application_1576797426048_311346 If using yarn, this is the yarn application id.
count Number of elements appended to the metrics buffer. 1 This value is usually 1, but not guaranteed to be constant. It is thread-safe.
endEpoch System time in milliseconds after resetting the metrics buffer. 1582504712723 The metrics buffer is an internal buffer used by this profiling application.
host Hostname of the executor. uber-bigdata-datanode2.uber.com
name Process name (number). 12345@uber-bigdata-datanode2.uber.com Includes the hostname as well.
processUuid UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743
processUuid_1 UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743 Appears to be a duplicate metric.
role Role of this spark executor. executor Can be either executor or driver.
stacktrace Stacktrace starting from bottom of the stacktrace. java.lang.Object.wait,java.lang.ref ... Trims the top method (most nested method) if the size is too large (800k characters)
startEpoch System time in milliseconds before resetting the metrics buffer. 1582504710523 The metrics buffer is an internal buffer used by this profiling application.
threadName Name of the thread for which stacktrace is displayed. Signal Dispatcher
threadState State of the thread when this stacktrace was captured. NEW, RUNNABLE, BLOCKED, WAITING ...

Back to Top

5. Thread Info

Name Description Sample Value Notes
time Time series value for grafana database. 1582504713009000000
epochMillis System time in milliseconds when this value was recorded. 1582504781508
host Hostname of the executor. uber-bigdata-datanode2.uber.com
name Process name (number). 12345@uber-bigdata-datanode2.uber.com Includes the hostname as well.
appId Application id for the job that is running. application_1576797426048_311346 If using yarn, this is the yarn application id.
processUuid UUID of the JVM process. 89a256c7-96c3-4e48-b5ba-f55e886ba743
role Role of this spark executor. executor Can be either executor or driver.
totalStartedThreadCount Total Number of Thread created so far since JVm Launch. 10
liveThreadCount Number of thread which are currently active. 8
newThreadCount Number of thread which are newly Created. 3 Derived from last Cycle Total Threads.
peakThreadCount the peak live thread count since the JVM start 8
Back to Top