Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Theano optimization failed #16

Open
domarps opened this issue May 27, 2018 · 1 comment
Open

Theano optimization failed #16

domarps opened this issue May 27, 2018 · 1 comment

Comments

@domarps
Copy link

domarps commented May 27, 2018

Hi,

I am trying to reproduce the code on a V100 instance and I ran into the following issues when I ran python NPE.py

Do you have any recommendations on how we can reproduce your experimental setup in the form of a Dockerfile?

/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file "/home/ubuntu/.config/matplotlib/matplotlibrc", line #2
  (fname, cnt))
/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file "/home/ubuntu/.config/matplotlib/matplotlibrc", line #3
  (fname, cnt))
Loading weights
Compiling Theano Functions
ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617198864 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>


Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison


ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617838416 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>


Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison


ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617837136 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>


Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison


ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu)
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2074, in process_node
    remove=remove)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
    fgraph.replace(r, new_r, reason=reason, verbose=False)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
    ". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
  Variable: id 139714617737296 CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}.0
  Op CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False}(Elemwise{Cast{float64}}.0, enc_conv1.W)
  Value Type: <type 'NoneType'>
  Old Value:  None
  New Value:  None
  Reason:  LocalOptGroup(local_abstractconv_gemm,local_abstractconv_gradweight_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gemm,local_abstractconv3d_gradweight_gemm,local_abstractconv3d_gradinputs_gemm,local_conv2d_cpu,local_conv2d_gradweight_cpu,local_conv2d_gradinputs_cpu). The type of the replacement must be the same.
  Old Graph:
  AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False} [id A] <TensorType(float32, 4D)> ''
   |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>

  New Graph:
  CorrMM{((2, 2), (2, 2)), (2, 2), (1, 1), 1 False} [id D] <TensorType(float64, 4D)> ''
   |Elemwise{Cast{float64}} [id E] <TensorType(float64, 4D)> ''
   | |X [id B] <TensorType(float32, 4D)>
   |enc_conv1.W [id C] <TensorType(float64, 4D)>


Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison


ERROR (theano.gof.opt): Optimization failure due to: local_abstractconv_check
ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode=(2, 2), subsample=(2, 2), filter_flip=False, imshp=(None, 3, 64, 64), kshp=(128, 3, 5, 5), filter_dilation=(1, 1), num_groups=1, unshared=False}(X, enc_conv1.W)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2034, in process_node
    replacements = lopt.transform(node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/tensor/nnet/opt.py", line 500, in local_abstractconv_check
    node.op.__class__.__name__)
LocalMetaOptimizerSkipAssertionError: AbstractConv2d Theano optimization failed: there is no implementation available supporting the requested options. Did you exclude both "conv_dnn" and "conv_gemm" from the optimizer? If on GPU, is cuDNN available and does the GPU support it? If on CPU, do you have a BLAS library installed Theano can link against? On the CPU we do not support float16.

Traceback (most recent call last):
  File "NPE.py", line 19, in <module>
    model = IAN(config_path = 'IAN_simple.py', dnn = False)
  File "/home/ubuntu/Neural-Photo-Editor/API.py", line 51, in __init__
    self.Z_hat_fn = theano.function([self.X],self.Z_hat)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
    output_keys=output_keys)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1839, in orig_function
    name=name)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1519, in __init__
    optimizer_profile = optimizer(fgraph)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 108, in __call__
    return self.optimize(fgraph)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 97, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 251, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 97, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2143, in apply
    nb += self.process_node(fgraph, node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 2039, in process_node
    lopt, node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 1933, in warn_inplace
    return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node)
  File "/home/ubuntu/anaconda3/envs/python2/lib/python2.7/site-packages/theano/gof/opt.py", line 1919, in warn
    raise exc
theano.gof.opt.LocalMetaOptimizerSkipAssertionError: AbstractConv2d Theano optimization failed: there is no implementation available supporting the requested options. Did you exclude both "conv_dnn" and "conv_gemm" from the optimizer? If on GPU, is cuDNN available and does the GPU support it? If on CPU, do you have a BLAS library installed Theano can link against? On the CPU we do not support float16.
@ajbrock
Copy link
Owner

ajbrock commented May 31, 2018

I've not used Docker at all (well, I've not successfully used Docker at all), but I can try to point you in the right direction. I was running this with python 2.7.12, Theano 0.7 or 0.8, and Lasagne 0.2 back in 2016, on GTX 980s and my laptop GPU for inference (GT730m).

At the end of the day though, I'm not sure you'll be able to run it on the V100s, what with Theano being deprecated and this code being 2+ years old at this point.

If you just want to use the interface, it should be trivial to plug in a PyTorch/Chainer/TF pre-trained GAN, just implement the appropriate methods in API.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@domarps @ajbrock