Profiling benchmark_cnn.py using Callgrind

[ShreyaMaheshwari]

On profiling the basic classification code, given on the tensorflow website (https://www.tensorflow.org/tutorials/keras/basic_classification), using callgrind, I get multiple errors as the following.

brk segment overflow in thread #1: can't grow to 0x4a3c000 (with different addresses)

Even thought I get these errors, the python program runs along with the profiler and gives me a profile data file named callgrind.out.pid, where pid is the process id of the file being profiled.

On the other hand, when trying to profile the benchmark_cnn.py file, the same brk segment overflow error occurs mutliple times (as mentioned above). But after the program has finished execution, no callgrind.out.pid file is seen.

Another strange thing is that I have observed is that the basic classification code takes around 30 seconds to run without callgrind and about 15 minutes with callgrind. At the same time, the benchmark_cnn.py file runs in around 2.5 hours normally (without profiler), but while doing it with callgrind, it gives me the output in a few minutes.

Why is the program running only for a short period of time? Why is no callgrind.out.pid produced?

[reedwm]

It looks this this error is fairly normal and does not indicate an issue in TensorFlow and tf_cnn_benchmarks.

I'm not familiar with callgrind so I'm not sure why there is no callgrind.out.pid. Perhaps callgrind can sometimes handle the segment overflow, but not always? As for why the command gives an output in a few minutes, are you sure the process is ending normally and not crashing?

A command to reproduce would be helpful, but I cannot guarentee I can help given that tf_cnn_benchmarks is unmaintained and I am not familiar with callgrind.