Installation w/ Miniconda, Reticulate 1.14, Tensorflow 2.0.0 & Keras 2.2.5.0 failing

[faltinl]

In connection with the new version of reticulate v1.14 and after quite a few problems to get Keras available, I encountered a rather serious problem. In an existing program written under keras v2.2.0.9000, reticulate v1.10 and tensorflow v1.9 and since then run successfully many times, the command

model %>% compile(
  optimizer = "adam",
  loss = list(loss_mean_squared_error, loss_categorical_crossentropy)
)

now run under reticulate 1.14, tensorflow 2.0.0 and keras 2.2.5.0, produce the following error message:

error in UseMethod ("compile") :
not applicable method 'compile' applied for object of class "c('keras._impl.keras.engine.training.Model', 'keras._impl.keras.engine.topology.Network', 'tensorflow.python.layers.network.GraphNetwork', 'keras._impl.keras.engine.topology.Layer', 'tensorflow.python.layers.base.Layer', 'python.builtin.object')"

Similarly, the command

model %>% fit(x = x,
              y = list(y1, y2),
              sample_weight = list(sample_weights, sample_weights),
              epochs = 10)

produces the error message

error in UseMethod ("fit") :
not applicable method 'compile' applied for object of class "c('keras._impl.keras.engine.training.Model', 'keras._impl.keras.engine.topology.Network', 'tensorflow.python.layers.network.GraphNetwork', 'keras._impl.keras.engine.topology.Layer', 'tensorflow.python.layers.base.Layer', 'python.builtin.object')"

So - which method would then be applicable for compilation? Consulting the keras 2.2.5.0 index w.r.t. command compile shows that compile as such doesn't exist as entry in the index any more, but only

compile.keras.engine.training.Model ... Configure a Keras model for training

However, under this symbolic headline I find the same description as earlier under keyword compile(...) and I cannnot detect anything wrong within the command used as above. Correspondingly the situation for fit().

Moreover, the function to_categorical(y, num_classes = NULL, ...) now seems to require specification of num_classes explicitely, since without this specification (as it was used up to now), a mysterious error message

Error: Keras loaded from tensorflow v1.5, however version 1.9 is required. Please update with tensorflow::install_tensorflow()

appears. And, really, the command tensorflow::tf_config() shows, much to my surprise,

TensorFlow v1.5.0 (C:...\Local\conda\conda\envs\R-TENS~1\lib\site-packages\tensorflow_init_.p)
Python v3.6 (C:/.../Local/conda/conda/envs/r-tensorflow/python.exe)

Note that in my case actually tensorflow v2.0.0 has been installed; the recommended command to install tensorflow v1.9 (sic!) at this point sounds like a dire reminiscence to my (lost) previous era with Keras. And there is no way back: an attempt to reinstall tensorflow 1.9 automatically leads to reinstallation of tensorflow 2.0.0, since tensorflow 1.9 is not available for R 3.6.2. Catch 362, so to speak...

I am really at a loss how to proceed. Any useful hint would be highly appreciated.

Tnx in advance,
Leo Faltin

[dfalbel]

It seems that you have the correct version of the R package but, an older version of the python package installed. Can you try running:

tensorflow::install_tensorflow(version="2.0.0")

This will install the 2.0.0 python package.

[faltinl]

Tnx for reply. I have tried that already and there were problems with R 3.6.2. So, this time, I downgraded to R 3.5.3 and started from scratch:

> devtools::install_github("rstudio/keras")

This installed keras 2.2.5.0, reticulate 1.14 tensorflow 2.0.0. The whole process was only interrupted by the prompt for installation of Miniconda and I did a restart R before having run the next line:

> tensorflow::install_tensorflow(version="2.0.0")

Again, everything run flawlessly and I did a restart R afterwards. Then, the usual sequence followed - unfortunately with a disappoiting result:

> library(reticulate) # 1.14
> use_condaenv("r-tensorflow", required = TRUE)
> library(tensorflow) # 2.0.0
> library(keras)      # 2.2.5.0
> is_keras_available()
[1] FALSE

The usual tests led to results as follows:

> sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)
Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252   
[3] LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] keras_2.2.5.0    tensorflow_2.0.0 reticulate_1.14 

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3        rstudioapi_0.10   whisker_0.4       magrittr_1.5     
 [5] rappdirs_0.3.1    usethis_1.5.1     devtools_2.2.1    pkgload_1.0.2    
 [9] R6_2.4.1          rlang_0.4.2       fansi_0.4.1       tools_3.5.3      
[13] pkgbuild_1.0.6    sessioninfo_1.1.1 cli_2.0.1         withr_2.1.2      
[17] tfruns_1.4        ellipsis_0.3.0    remotes_2.1.0     assertthat_0.2.1 
[21] digest_0.6.23     rprojroot_1.3-2   crayon_1.3.4      processx_3.4.1   
[25] callr_3.4.0       base64enc_0.1-3   fs_1.3.1          ps_1.3.0         
[29] zeallot_0.1.0     testthat_2.3.1    memoise_1.1.0     glue_1.3.1       
[33] compiler_3.5.3    generics_0.0.2    desc_1.2.0        backports_1.1.5  
[37] prettyunits_1.1.0 jsonlite_1.6  

> reticulate::py_config()
python:         C:/.../Local/conda/conda/envs/r-tensorflow/python.exe
libpython:      C:/.../Local/conda/conda/envs/r-tensorflow/python36.dll
pythonhome:     C:/.../Local/conda/conda/envs/r-tensorflow
version:        3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/.../Local/conda/conda/envs/r-tensorflow/Lib/site-packages/numpy
numpy_version:  1.14.3
tensorflow:     C:\...\Local\conda\conda\envs\R-TENS~1\lib\site-packages\tensorflow\__init__.p
NOTE: Python version was forced by use_python function

> tensorflow::tf_config()
TensorFlow v1.5.0 (C:\...\Local\conda\conda\envs\R-TENS~1\lib\site-packages\tensorflow\__init__.p)
Python v3.6 (C:/.../Local/conda/conda/envs/r-tensorflow/python.exe)

> implementation()
Module(tensorflow.keras)

> keras.backend.backend()
Error in keras.backend.backend(): could not find function "keras.backend.backend"

Thus, while tensorflow is working as it should, keras itself has problems. And since I didn't find any indications as to the reson for this behaviour whatsoever, I really need help. Thanks in advance...

[dfalbel]

is_keras_available() is not the function you should use to check if your installation is working. It checks if the Keras python package is available, but it's not necessary in order to the Keras package to work because we are using the TensorFlow implementation.

The problem here is that for some reason you are still getting the TensorFlow v1.5 in your environment.
Can you try use_condaenv("r-reticulate", required = TRUE) Note, the r-reticulate instead of r-tensorflow.

[faltinl]

Tnx for suggestions. If after tensorflow::install_tensorflow(version="2.0.0") I do a restart R and then

library(reticulate) 
use_condaenv("r-reticulate", required = TRUE)  [Note: same test-results WITHOUT this line]
library(tensorflow)
library(keras)     

I invariably get the following results for the tests:

> reticulate::py_config()
python:         C:/Users/test/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython:      C:/Users/test/AppData/Local/r-miniconda/envs/r-reticulate/python36.dll
pythonhome:     C:/Users/test/AppData/Local/r-miniconda/envs/r-reticulate
version:        3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/test/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.18.1
tensorflow:     C:\Users\test\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\tensorflow\__init__.p
NOTE: Python version was forced by use_python function

But then for tensorflow:

> tensorflow::tf_config()
Installation of TensorFlow not found.
Python environments searched for 'tensorflow' package:
C:\Users\test\AppData\Local\r-miniconda\envs\r-reticulate\python.exe
You can install TensorFlow using the install_tensorflow() function.

This changes into the message that tensorflow has been downgraded to v1.5 only in case of use_condaenv("r-tensorflow", required = TRUE). And finally an error message after the line

> library(keras)  
Error: package or namespace load failed for ‘keras’:
 .onLoad in loadNamespace() for 'keras' failed, details:
  Call: py_module_import(module, convert = convert)
  Error: ImportError: Traceback (most recent call last):
  File "C:\Users\test\AppData\Local\r-miniconda\envs\r-reticulate\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\test\AppData\Local\r-miniconda\envs\r-reticulate\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\test\AppData\Local\r-miniconda\envs\r-reticulate\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Users\test\AppData\Local\r-miniconda\envs\r-reticulate\lib\imp.py", line  

...which I don't understand at all, but which also doesn't appear in case of use_condaenv("r-tensorflow", required = TRUE).

For your information: Since my last attempts I have removed completely R3.5.3, Miniconda, RStudio1.2.5033 (including a thorough clean-up of all associated directories) and reinstalled R3.6.1, and again RStudio1.2.5033 as well as Miniconda by means of tensorflow::install_tensorflow(version="2.0.0"), but apparently to no avail - I'm really desperate by now.

[skeydan]

For background: Ana-/Miniconda manages Python environments for you, where each environment can have completely different Python libraries.

So, please forget about the old r-tensorflow, and just concentrate on r-reticulate, which now has become the name of the default environment reticulate creates for you when/if it installs Miniconda.

Also, the reason to always use required=TRUE is that if you don't, you won't know if it actually happened or not. (It can only happen if Python is initialized afresh, which normally only will happen if you restart the R session.)

Now, the easiest way to do some further testing would probably involve the command line.
Normally,

conda activate r-reticulate

would "load" the respective environment. Then if you do

python

you should get a Python shell, and you should be able to try

import tensorflow as tf

Additionally - even before executing python

you could do

pip list

to see a list of installed libraries. Can you let us know what that yields?

[faltinl]

Hi, tnx a lot for the very useful explanation. import tensorflow as tf got me a lengthy error message. Since I've never used python / python shell: how do I copy this message? Copy/Paste obviously doesn't work, sry.
In the meantime I include a screen shot.

[faltinl]

Here is the pip list, again as a PNG...

[skeydan]

So tensorflow isn't there... can you do - right there on the command line, after

conda activate r-reticulate

:

pip install tensorflow

and if that fails, paste the error message, otherwise try importing again?

[faltinl]

OK, seems to have worked, tensorflow 2.1.0, though.

[skeydan]

yeah, that's the current releasè since last week

if you need 2.0, do

pip uninstall tensorflow
pip install tensorflow==2.0

[faltinl]

Well, in the meantime I got this with tf2.1.0:

Don't really understand why it shouldn't have worked - the installed components are all well listed:

[faltinl]

My last two screen shots have perhaps been taken a bit too early. I had another go at import tensorflow as tf and got an error message very similar (except for a few, but definitely not all indicated line numbers) to the first one:

[faltinl]

Strange enough: I did another pip list which shows: tensorflow 2.1.0 is definitely there (only the area with the tf-components is shown):

So - why then is it not imported?

Observed error traces look similar to error report #22794, "ImportError: DLL load failed: The specified module could not be found" - with even several code line numbers coninciding. Case #22512 seems to go into a remotely similar direction.

[Srinivasan2017]

am also facing same issues .. Kindly help..

[faltinl]

Whatever I do (I have changed to Anaconda3 w/ Py3.6.10, in the meantime), I end up with

> reticulate::py_config()
python:         C:/Anaconda/envs/r-reticulate/python.exe
libpython:      C:/Anaconda/envs/r-reticulate/python36.dll
pythonhome:     C:/Anaconda/envs/r-reticulate
version:        3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Anaconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.18.1
tensorflow:     C:\Anaconda\envs\R-RETI~1\lib\site-packages\tensorflow\__init__.p
NOTE: Python version was forced by use_python function

and then - although install_tensorflow() was indicated to have worked correctly every times I applied it -

> tensorflow::tf_config()
Installation of TensorFlow not found.
Python environments searched for 'tensorflow' package:
 C:\Anaconda\envs\r-reticulate\python.exe
You can install TensorFlow using the install_tensorflow() function.

What can I do to get Tensorflow in working condition? Not to forget that I still need to get Keras working too...

Please help.

[skeydan]

Not to forget that I still need to get Keras working too...

On the upside, Keras is part of TF, there'll be nothing to do there ;-)

One of the issues you're linking to hints to the problem possibly being linked to Windows GPU binaries now requiring "Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019".

To check if this is the issue, can you do

conda activate r-reticulate
pip uninstall tensorflow
pip install tensorflow-cpu

For background, since 2.1 the default binary is the GPU-enabled one; the CPU-only one can be installed separately.

Then when you test, please make sure you're in the right environment (r-reticulate) always.

[faltinl]

Tnx for reply & suggestions which I understand quite well. However, since I did so many experiments (as I would rather call them by now), I had already removed Tensorflow 2.1.0 in the meantime. So the problems observed apparently have nothing to do with either 2.1.0 nor 2.0.0. And rightly so: after installing tensorflow-cpu as indicated above, an error message like the last one appeared again. So essentially, nothing has changed, I have tensorflow 2.1.0 again in my pip list; RStudio package list, however, still shows 2.0.0. btw. Anyway, tensorflow::tf_config() still shows Installation of TensorFlow not found.

At the moment I suspect my problems have to do with the path leading to tensorflow. I found an issue dealing with that, where quite similar problems like mine disappeard after doing the tensorflow-installation with

install_tensorflow(method="conda",conda=[path])

but: how do I find the [path] when even the programme itself doesn't find it? Simply doing install_tensorflow(method="conda") doesn't help anyway, I have tried that one already...

PS: Oh, yes - and the "Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019" I have already installed, too, in the meantime.

[skeydan]

I had already removed Tensorflow 2.1.0 in the meantime. So the problems observed apparently have nothing to do with either 2.1.0 nor 2.0.0. And rightly so: after installing tensorflow-cpu as indicated above, an error message like the last one appeared again. So essentially, nothing has changed, I have tensorflow 2.1.0 again in my pip list;

I do not follow ... You removed 2.1 yet pip says you have it?

At this point, I can only suggest the same as above

conda activate r-reticulate
pip uninstall tensorflow
pip install tensorflow-cpu
python 
> import tensorflow as tf

Alternatively:

conda activate r-reticulate
pip uninstall tensorflow
pip install tensorflow=2.0
python 
> import tensorflow as tf

We first need to get it working from Python, and then see about R.

tensorflow/tensorflow#35749
tensorflow/tensorflow#35618

[faltinl]

Yes, as I said: After our attempts from 4 days ago with tensorflow 2.1.0 I deinstalled that, then repeated the process with 2.0.0, a second time with 2.1.0 and a second time with 2.0.0. And from this last attempt obviously remained one tensorflow 2.1.0, together with the last installation of tensorflow 2.0.0, see pip list from just a few mins after I received your last post:

Since there was an apparently already complete set of tensorflow 2.0.0 I decided to follow your alternative, uninstalled with pip uninstall tensorflow the default version, i.e. tensorflow 2.1.0, and installed pip install tensorflow==2.0. This produced a long trace saying for every component Requirement already satisfied (everything was there already, as expected), and so I tried import tensorflow as tf under python. This however, didn't work, see pic:

Apparently, something went wrong with this "partial" uninstall/install actions or so, don't know. I therefore uninstalled tensorflow 2.0.0 (i.e. definitely both versions were then absent), and finally did pip install tensorflow==2.0 again, successfully as it said. Changing to python shell again and trying to import tensorflow as tf again produced the well known long error trace from a few posts above.

From within RStudio nothing has changed either, as the command sequence

library(reticulate) 
use_condaenv("r-reticulate", required = TRUE)
library(tensorflow) 

again gives

> tensorflow::tf_config()
Installation of TensorFlow not found.
...

[faltinl]

In the meantime I changed from R3.6.1 (when installing R-packages, I got several warnings that they have been built under R3.6.2) to R3.6.2 and from Anaconda3 to Miniconda3 - to no avail: Not only did I get the same message from tensorflow::tf_config() as above, but also trying to import tensorflow as tf (v2.0.0) under python led to the same lengthy error message shown further above.

Against much earlier attemps the situation has changed insofar, as I now keep getting identical error messages consistently (i.e. since about 10 days intensive trials). This indicates, that there must be a rather basic problem, common to all the undertaken attempts, behind this installation problem. As I have not the slightest idea what that could be, I'm asking desperately for help.

[skeydan]

I don't like saying this, but given that your issue looks so similar to the ones I linked in my comment above (and I think when you say, you get one error message again and again, it's the DLL not found one, right?), while these two issues were closed in TF either with downgrade to 2.0 or installing that Visual Studio thing ... I think it makes most sense for you to create an issue in the TF repo directly.

They might have an idea which DLL is not found, or how to find out...

Sorry for not having better advice in this situation...

If you create that issue in TF, please link it here so we can follow.

[faltinl]

Tnx. The issue was forwarded to Tensorflow 2.0.0 cpu, import error: DLL load failed #36138.

[skeydan]

Ok, let's hope for the best :-)

[skeydan]

BTW you may want to edit the issue and remove anything R/RStudio-related, as it is not relevant and could only be distracting :-)

[faltinl]

I thought so too, but looking at the actual install guides makes me nervous: if the so called 'simple' and 'quick' installation via RStudio should really not be possible (it looks as if nobody seems to be interested in that any more), I would have to forget my running ML projects for which I have worked the past 3 years for quite a long time. To learn Python and the handling of several auxiliary 'environments' etc. just for being able to bring tensorflow back to life on my computer is no fun for people like me who are interested in applications and not in IT itself...

[skeydan]

I should have been more clear - what reticulate / tensorflow do is nothing than install TF for you; they do exactly the same thing you ended up doing with Python on the command line.

But if an error occurs that is directly attributable to the TF installation per se (not to R not managing to find it, say), all that is needed to diagnose is information about the Python installation.

Even if the problem has nothing to do with R, we always try to help. But in some cases, like yours, it may make sense to file an issue in the original repository. Especially since your issue is so similar to ones that were closed with solutions none of which worked for you.

(BTW, added a comment to your issue in that respect.)