Add possibility to specify number of threads in PyKeras and set specific Tensorflow session config options #2566

lmoneta · 2018-08-31T12:49:28Z

Add a new option in PyKeras to select the number of running threads when running keras with tensorflow.

If nothing is specified, tensor flow normally runs by default on all available cores

phsft-bot · 2018-08-31T12:49:41Z

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dccache=ON
How to customize builds

stwunsch · 2018-09-16T11:17:22Z

tmva/pymva/src/MethodPyKeras.cxx

+   if (num_threads > 0) { 
+      PyRunString("import tensorflow as tf");
+      PyRunString("from keras.backend import tensorflow_backend as K");
+      PyRunString("print 'setting tensorflow to run as single thread' "); 


This breaks for Theano or CNTK as backend!

I agree, I will add an if statement that this will work only in case of Tensorflow

phsft-bot · 2018-10-01T12:57:02Z

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dccache=ON
How to customize builds

phsft-bot · 2018-10-01T12:59:37Z

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dccache=ON
How to customize builds

phsft-bot · 2018-10-01T13:07:49Z

Build failed on ubuntu16/native.
See console output.

phsft-bot · 2018-10-01T13:17:23Z

Build failed on centos7/clang39.
See console output.

Warnings:

/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-clang39-opt/lib/cmake/VecCore/../../../include/Vc/scalar/../common/../sse/intrinsics.h:616:16: warning: variable 'f' may be uninitialized when used here [-Wconditional-uninitialized]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-clang39-opt/lib/cmake/VecCore/../../../include/Vc/scalar/../common/memorybase.h:463:83: warning: unused parameter 'flags' [-Wunused-parameter]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-clang39-opt/lib/cmake/VecCore/../../../include/Vc/scalar/../common/memorybase.h:478:89: warning: unused parameter 'flags' [-Wunused-parameter]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-clang39-opt/lib/cmake/VecCore/../../../include/Vc/common/simdize.h:913:1: warning: 'tuple_size' defined as a class template here but previously declared as a struct template [-Wmismatched-tags]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-clang39-opt/lib/cmake/VecCore/../../../include/Vc/common/simdize.h:920:1: warning: 'tuple_element' defined as a class template here but previously declared as a struct template [-Wmismatched-tags]

phsft-bot · 2018-10-01T13:20:10Z

Build failed on centos7/gcc62.
See console output.

phsft-bot · 2018-10-01T13:26:41Z

Build failed on centos7/gcc7.
See console output.

Warnings:

../root/core/newdelete/src/NewDelete.cxx:217:11: warning: ‘operator new’ must not return NULL unless it is declared ‘throw()’ (or -fcheck-new is in effect)
../root/core/newdelete/src/NewDelete.cxx:344:11: warning: ‘operator new’ must not return NULL unless it is declared ‘throw()’ (or -fcheck-new is in effect)
../root/core/newdelete/src/NewDelete.cxx:263:6: warning: the program should also define ‘void operator delete(void*, std::size_t)’ [-Wsized-deallocation]
../root/core/newdelete/src/NewDelete.cxx:367:6: warning: the program should also define ‘void operator delete [](void*, std::size_t)’ [-Wsized-deallocation]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-gcc7-opt/include/python2.7/unicodeobject.h:534:24: warning: ISO C++1z does not allow ‘register’ storage class specifier [-Wregister]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-gcc7-opt/include/python2.7/unicodeobject.h:553:24: warning: ISO C++1z does not allow ‘register’ storage class specifier [-Wregister]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-gcc7-opt/include/python2.7/unicodeobject.h:575:29: warning: ISO C++1z does not allow ‘register’ storage class specifier [-Wregister]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-gcc7-opt/include/python2.7/unicodeobject.h:593:23: warning: ISO C++1z does not allow ‘register’ storage class specifier [-Wregister]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-gcc7-opt/include/python2.7/stringobject.h:173:24: warning: ISO C++1z does not allow ‘register’ storage class specifier [-Wregister]
/cvmfs/sft.cern.ch/lcg/views/LCG_rootext20180517/x86_64-centos7-gcc7-opt/include/python2.7/stringobject.h:174:21: warning: ISO C++1z does not allow ‘register’ storage class specifier [-Wregister]

And 314 more

phsft-bot · 2018-10-01T13:34:08Z

Build failed on slc6/gcc48.
See console output.

phsft-bot · 2018-10-01T13:34:36Z

Build failed on slc6-i686/gcc49.
See console output.

phsft-bot · 2018-10-01T13:37:46Z

Build failed on fedora28/native.
See console output.

Errors:

/mnt/build/workspace/root-pullrequests-build/root/tmva/pymva/src/MethodPyKeras.cxx:148:34: error: ‘PyString_AsString’ was not declared in this scope

Warnings:

/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<clang::CXXRecordDecl*, clang::SourceLocation>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<const clang::CXXMethodDecl*, const clang::CXXMethodDecl*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<clang::CXXMethodDecl*, const clang::FunctionProtoType*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<clang::CXXRecordDecl*, clang::SourceLocation>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<const clang::CXXMethodDecl*, const clang::CXXMethodDecl*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]

And 68 more

phsft-bot · 2018-10-01T14:35:06Z

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dccache=ON
How to customize builds

phsft-bot · 2018-10-01T14:38:02Z

Build failed on fedora28/native.
See console output.

Errors:

/mnt/build/workspace/root-pullrequests-build/root/tmva/pymva/src/MethodPyKeras.cxx:93:34: error: ‘PyString_AsString’ was not declared in this scope

Warnings:

/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<clang::CXXRecordDecl*, clang::SourceLocation>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<const clang::CXXMethodDecl*, const clang::CXXMethodDecl*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<clang::CXXMethodDecl*, const clang::FunctionProtoType*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:309:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<void*, long unsigned int>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<clang::CXXRecordDecl*, clang::SourceLocation>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:296:13: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘struct std::pair<const clang::CXXMethodDecl*, const clang::CXXMethodDecl*>’ with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]

And 68 more

phsft-bot · 2018-10-05T13:09:40Z

Build failed on windows10/vc15.
See console output.

Errors:

error: failed to execute prompt script (exit code 1)

phsft-bot · 2019-02-05T11:11:56Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

lmoneta · 2019-02-05T11:20:21Z

Added also possibility to set gpu_options.
Added options session.gpu_options.allow_growth= True neded for running cuDNN on new RTX cards
See https://www.tensorflow.org/guide/using_gpu#allowing_gpu_memory_growth
and solution described here for RTX cards:
tensorflow/tensorflow#24496

ashlaban

Looks good to me! Just a few minor discussion points on style.

ashlaban · 2019-03-01T11:47:18Z

tmva/pymva/src/MethodPyKeras.cxx

+   //  -  set up number of threads for CPU if NumThreads option was specified
+
+   // check first if using tensorflow backend
+   PyRunString("keras_backend_is_tf =  keras.backend.backend() == \"tensorflow\"");


This could be refactored into a separate function TMVA::PyKeras::kBackends GetKerasBackend(); to raise the level of abstraction a little bit.

(Possibly just as a named lambda function.)

ashlaban · 2019-03-01T11:48:49Z

tmva/pymva/src/MethodPyKeras.cxx

+      PyRunString("import tensorflow as tf");
+      PyRunString("from keras.backend import tensorflow_backend as K");
+      // in case specify number of threads
+      int num_threads = fNumThreads;


Remove this and just use fNumThreads directly?

ashlaban · 2019-03-01T11:49:41Z

tmva/pymva/src/MethodPyKeras.cxx

+      else
+         PyRunString("session_conf = tf.ConfigProto()");
+
+      PyRunString("session_conf.gpu_options.allow_growth = True"); // for GPU's (needed for new RTX cards)


I will just mention this for reference:
As you mentioned during our discussion today this could be extracted to an option.

ashlaban · 2019-03-01T11:49:57Z

tmva/pymva/src/MethodPyKeras.cxx

   // Setup model, either the initial model from `fFilenameModel` or
   // the trained model from `fFilenameTrainedModel`
   if (fContinueTraining) Log() << kINFO << "Continue training with trained model" << Endl;
   SetupKerasModel(fContinueTraining);
+


Should this be here?

… tensorflow

…e to the right place

…l config option for GPU that is needed for new RTX cards

Define a new options GpuOPtions to allow to confire the Gpu when using Tensorflow. FOr example, to minimize GPu emory usage, needed for running Keras and then TMVA on GPU, you can specify: "GpuOptions=allow_growth=True"

phsft-bot · 2019-03-08T13:46:45Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

amadio assigned lmoneta Sep 3, 2018

stwunsch reviewed Sep 16, 2018

View reviewed changes

lmoneta force-pushed the pymva-add-nthreads-opt branch from 07537e3 to 1b59336 Compare October 1, 2018 12:59

lmoneta force-pushed the pymva-add-nthreads-opt branch from 9c8f3c0 to 96504dc Compare February 5, 2019 11:11

lmoneta requested a review from ashlaban as a code owner February 5, 2019 11:11

lmoneta changed the title ~~Add possibility to specify number of threads in PyKeras~~ Add possibility to specify number of threads in PyKeras and set specific Tensorflow session config options Feb 5, 2019

ashlaban approved these changes Mar 1, 2019

View reviewed changes

Lorenzo Moneta and others added 6 commits March 7, 2019 18:20

Add an option to select the number of threads when running keras with…

9be0c3a

… tensorflow

add a check that backend is tensorflow

b1210cc

Fix setting the number of threads in PyKeras/Tensorflow. Move the cod…

de5009b

…e to the right place

Fix setting number of threads for tensorflow backend. Add also specia…

94be999

…l config option for GPU that is needed for new RTX cards

Improve printout of log messages

218f8c3

lmoneta force-pushed the pymva-add-nthreads-opt branch from 96504dc to 218f8c3 Compare March 8, 2019 13:46

lmoneta merged commit 6897056 into root-project:master Mar 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add possibility to specify number of threads in PyKeras and set specific Tensorflow session config options #2566

Add possibility to specify number of threads in PyKeras and set specific Tensorflow session config options #2566

lmoneta commented Aug 31, 2018

phsft-bot commented Aug 31, 2018

stwunsch Sep 16, 2018

lmoneta Sep 28, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 5, 2018

phsft-bot commented Feb 5, 2019

lmoneta commented Feb 5, 2019

ashlaban left a comment

ashlaban Mar 1, 2019

ashlaban Mar 1, 2019

ashlaban Mar 1, 2019

ashlaban Mar 1, 2019

phsft-bot commented Mar 8, 2019

Add possibility to specify number of threads in PyKeras and set specific Tensorflow session config options #2566

Add possibility to specify number of threads in PyKeras and set specific Tensorflow session config options #2566

Conversation

lmoneta commented Aug 31, 2018

phsft-bot commented Aug 31, 2018

stwunsch Sep 16, 2018

Choose a reason for hiding this comment

lmoneta Sep 28, 2018

Choose a reason for hiding this comment

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

Warnings:

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

Warnings:

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

Errors:

Warnings:

phsft-bot commented Oct 1, 2018

phsft-bot commented Oct 1, 2018

Errors:

Warnings:

phsft-bot commented Oct 5, 2018

Errors:

phsft-bot commented Feb 5, 2019

lmoneta commented Feb 5, 2019

ashlaban left a comment

Choose a reason for hiding this comment

ashlaban Mar 1, 2019

Choose a reason for hiding this comment

ashlaban Mar 1, 2019

Choose a reason for hiding this comment

ashlaban Mar 1, 2019

Choose a reason for hiding this comment

ashlaban Mar 1, 2019

Choose a reason for hiding this comment

phsft-bot commented Mar 8, 2019