Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report driver memory statistics with QuboleJobListener #37

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

mayurdb
Copy link
Collaborator

@mayurdb mayurdb commented May 23, 2019

Following metrics related to the Driver are now reported -

  1. driverHeapMax => Max Heap memory allocated to the driver JVM
  2. driverMaxHeapCommitted => Max Heap memory committed to the driver JVM
  3. driverMaxHeapUsed => Max Heap memory used by the driver JVM
  4. driverCPUTime => Total CPU Time allocated to the driver JVM
  5. driverGCCount => Total major GCs happened during the lifetime of the JVM
  6. driverGCTime => Part of driverCPUTime spent in the GC

Sample output for a simple application.


 Time spent in Driver vs Executors
 Driver WallClock Time    00m 11s   79.24%
 Executor WallClock Time  00m 03s   20.76%
 Total WallClock Time     00m 15s


 DriverMetrics (driverMetrics)
                NAME                        Value
 driverCPUTime                           10216.7 hh
 driverGCCount                                   11
 driverGCTime                              301.0 ms
 driverHeapMax                               1.0 GB
 driverMaxHeapCommitted                    480.0 MB
 driverMaxHeapUsed                         308.9 MB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant