How To Get a Ruby Heap Dumps & GC Stats from CC

1. Apply the following patch:

  From 7ca3950418af73670ba33ed3e0c78606e06c7b55 Mon Sep 17 00:00:00 2001
  From: Raina Masand <rmasand@pivotal.io>
  Date: Fri, 2 Mar 2018 15:01:59 -0800
  Subject: Patch to make heap dump possible
  
  Signed-off-by: Tim Downey <tdowney@pivotal.io>
  ---
   lib/cloud_controller/runner.rb | 19 +++++++++++++++++++
   1 file changed, 19 insertions(+)
  
  diff --git a/lib/cloud_controller/runner.rb b/lib/cloud_controller/runner.rb
  index 03895fa..032ab57 100644
  --- a/lib/cloud_controller/runner.rb
  +++ b/lib/cloud_controller/runner.rb
  @@ -13,6 +13,7 @@ require 'cloud_controller/metrics/periodic_updater'
   require 'cloud_controller/metrics/request_metrics'
   
   require_relative 'message_bus_configurer'
  +require 'objspace'; ObjectSpace.trace_object_allocations_start
   
   module VCAP::CloudController
     class Runner
  @@ -122,6 +123,24 @@ module VCAP::CloudController
           EM.add_timer(0) do
             logger.warn('Collecting diagnostics')
             collect_diagnostics
  +
  +          File.open("/tmp/heap_dump_before_gc", "w") do |file|
  +            ObjectSpace.dump_all(output: file)
  +          end
  +
  +          File.open("/tmp/gc_stat_before_gc", "w") do |file|
  +            file.write(GC.stat)
  +          end
  +
  +          GC.start
  +
  +          File.open("/tmp/heap_dump_after_gc", "w") do |file|
  +            ObjectSpace.dump_all(output: file)
  +          end
  +
  +          File.open("/tmp/gc_stat_after_gc", "w") do |file|
  +            file.write(GC.stat)
  +          end
           end
         end
       end
  -- 
  2.10.0

1. Restart the CC
2. Find the CC's pid at /var/vcap/sys/run/cloud_controller_ng/cloud_controller_ng.pid
3. kill -USR1 <cc's pid>
4. This will generate /tmp/heap_dump_before_gc, /tmp/gc_stat_before_gc, /tmp/heap_dump_after_gc, and /tmp/gc_stat_after_gc
5. You can use heapy to help analyze the resulting heap dumps