Using Caliper with Cray compiler

[mastino]

Hello,

I am trying to measure an OpenMP application with Caliper (from git on Nov 18 2020). The Caliper build and the benchmark are built with CCE 10.2. When the code runs with Caliper instrumentation it hangs and does not finish. Using the gcc compiler I have not had these problems. Do you have recommendations for debugging this issue?

Thanks!

[daboehme]

Hi @mastino , can you run the program with CALI_LOG_VERBOSITY=2 set as an environment variable? That should give us debug output that might tell us where it hangs. Also, you can try running the program in a debugger and try to get a stack trace once it hangs.

[mastino]

@jrmadsen I have an attribute created before the omp region, is that not sufficient? Is there still a cali_init() function? I can't seem to find it in the documentation

[jrmadsen]

I have had a lot of issues in the past like this when the first instance of a timemory component happens on a child thread, especially when all the measurements only happen on child threads and especially with non-GNU and non-Clang compilers since the ctor attribute wasn't necessarily supported. I know Caliper has the same sort of issues with static lazy initialization (@daboehme maybe someday we'll figure out how to do it 100% properly) but I've resolved a ton of my issues once I replaced relying on the ctor attribute with something like:

// void library_init_function() __attribute__((constructor));

void library_init_function();

namespace
{
// use comma operator to execute library_init_function() when
// the static initialization of _library_init_variable happens
static bool _library_init_variable = (library_init_function(), true);
}

[jrmadsen]

@mastino Yeah, try calling cali_init() although I would guess that the attribute creation would initialize caliper but @daboehme would obviously know better than me. I just know that @daboehme and I have struggled a lot with TLS initialization bc it is really, really difficult to robustly configure an implicit and lazy initialization system where data collection happen in thread-local memory (to avoid locking) and adds itself into parent thread. If cali_init() doesn't fix it, I'd suggest (as a debug exercise) seeing it creating a region on the main thread fixes it -- that would provide the evidence that this indeed another one of the TLS initialization issues.

[jrmadsen]

e.g.:

CALI_MARK_BEGIN("dummy");
puts("dummy");
CALI_MARK_END("dummy");

#pragma omp parallel private(iii,jjj,ii,jj,i,j,k)
{
#ifdef USE_CALI_REG
CALI_MARK_BEGIN("block_mm");
#endif

  #pragma omp for
  for (int i = 0; i < rows_a; i++ ) {
    for (int jjj = 0; jjj < cols_b; jjj = jjj + BLOCK_ROWS) 
      for (int j = jjj; j < min(cols_b, jjj + BLOCK_ROWS); j++) 
        for (int kkk = 0; kkk < cols_a; kkk = kkk + BLOCK_COLS) {
          #pragma omp simd
          // #pragma omp simd aligned(mat_a, mat_b, mat_c: 64)
          for (int k = kkk; k < min(cols_a,kkk + BLOCK_COLS); k++)
            mat_c[i][j] = mat_c[i][j] + mat_a[i][k] * mat_b[j][k];
        }
  } // i

#ifdef USE_CALI_REG
CALI_MARK_END("block_mm");
}

[mastino]

Thanks for the help; I tried with cali_init and with the dummy region, in both cases it runs fine the first time and then either seg faults or hangs. When it completes the first time it still has the distorted output as I posted above.

We have the newer cray compilers with clang so I will try with those as well sometime this week and provide an update.

[daboehme]

The create_attribute() does indeed initialize Caliper, so that shouldn't be the problem per se. There are indeed some issues with initializing Caliper in a multi-threaded region, but the create_attribute() should take care of that in this case. Creating a dummy region outside the parallel region first as @jrmadsen suggested is worth a try. There still seems to be some problem with the TLS though, given that we also have the crash in the TLS object destructor.