Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Full batch models fail to save with "If specifying TensorSpec names for nested structures, either zero or all names have to be specified" #1251

Open
huonw opened this issue Apr 16, 2020 · 7 comments · May be fixed by #1779
Open

Full batch models fail to save with "If specifying TensorSpec names for nested structures, either zero or all names have to be specified" #1251

huonw opened this issue Apr 16, 2020 · 7 comments · May be fixed by #1779
Assignees
@huonw
Labels
bug Something isn't working sg-library

Comments

@huonw
Copy link
Member

huonw commented Apr 16, 2020 •
edited

Description

As reported in https://community.stellargraph.io/t/valueerror-if-specifying-tensorspec-names-for-nested-structures-either-zero-or-all-names-have-to-be-specified/97, saving some models in sparse mode fails with ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',). This is due to a bug in Tensorflow tensorflow/tensorflow#38465 related to sparse tensors being passed between Keras layers. This bug is fixed in TensorFlow nightly (tf-nightly package) and the 2.3.0-rc0 release candidate.

Affected models (with sparse=True in their generators):

  • APPNP
  • GAT
  • GCN
  • RGCN
  • Deep Graph Infomax with any of the above

Work-around

  • 2.3.0-rc0 (recommended): replace tensorflow with the next release candidate: pip install tensorflow==2.3.0-rc0.
  • nightly: replace tensorflow with tf-nightly, with just pip install tf-nightly.

Once the 2.3 release is out, either of these should be able to be replaced with pip install tensorflow==2.3.0 (the nightly version may need an explicit pip uninstall tf-nightly tf-estimator-nightly tb-nightly).

Original bug:

Describe the bug

As reported in https://community.stellargraph.io/t/valueerror-if-specifying-tensorspec-names-for-nested-structures-either-zero-or-all-names-have-to-be-specified/97, saving GAT or any other full batch model seems to fail.

To Reproduce

import stellargraph as sg
import tensorflow as tf

def test(x):
    print(f"#### {type(x).__name__}")
    print("```")
    model = tf.keras.Model(*x.in_out_tensors())
    try:
        tf.keras.models.save_model(model, "/tmp")
    except Exception as e:
        print(f"tf.keras.models.save_model: {e!r}")
        
    try:
        tf.saved_model.save(model, "/tmp")
    except Exception as e:
        print(f"tf.saved_model.save: {e!r}")

    try:
        model.save("/tmp")
    except Exception as e:
        print(f"model.save: {e!r}")
    print("```")


G, _ = sg.datasets.Cora().load()

fb_gen = sg.mapper.FullBatchNodeGenerator(G)
test(sg.layer.GAT([4], fb_gen))

test(sg.layer.GCN([4], fb_gen))

test(sg.layer.PPNP([4], fb_gen, ["relu"]))
test(sg.layer.APPNP([4], fb_gen, ["relu"]))

gs_gen = sg.mapper.GraphSAGENodeGenerator(G, 5, [2])
test(sg.layer.GraphSAGE([4], gs_gen))

hs_gen = sg.mapper.HinSAGENodeGenerator(G, 5, [2], head_node_type="paper")
test(sg.layer.HinSAGE([4], hs_gen))

cg_gen = sg.mapper.ClusterNodeGenerator(G)
test(sg.layer.ClusterGCN([4], ["relu"], cg_gen))

av_gen = sg.mapper.Attri2VecNodeGenerator(G, 10)
test(sg.layer.Attri2Vec([4], av_gen))

rg_gen = sg.mapper.RelationalFullBatchNodeGenerator(G)
test(sg.layer.RGCN([4], rg_gen))

kg_gen = sg.mapper.KGTripleGenerator(G, 10)
test(sg.layer.ComplEx(kg_gen, 10))
test(sg.layer.DistMult(kg_gen, 10))

pg_gen = sg.mapper.PaddedGraphGenerator([G])
test(sg.layer.GCNSupervisedGraphClassification([4], ["relu"], pg_gen))

test(sg.layer.DeepGraphInfomax(sg.layer.GCN([4], fb_gen)))

test(sg.layer.DeepGraphInfomax(sg.layer.GraphSAGE([4], gs_gen)))

Observed behavior

The full batch methods (GAT, GCN, PPNP, APPNP and DeepGraphInfomax using any of these) fails with:

ValueError: If specifying TensorSpec names for nested structures, either zero or all names have to be specified.

(RGCN fails too, but that's a separate issue: #1252.)

Full stack trace 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-60-050090835af0> in <module>
     27 
     28 fb_gen = sg.mapper.FullBatchNodeGenerator(G)
---> 29 test(sg.layer.GAT([4], fb_gen))
     30 
     31 test(sg.layer.GCN([4], fb_gen))

<ipython-input-60-050090835af0> in test(x)
      7     model = tf.keras.Model(*x.in_out_tensors())
      8     try:
----> 9         tf.keras.models.save_model(model, "/tmp")
     10     except Exception as e:
     11         print(f"tf.keras.models.save_model: {e!r}")

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options)
    113   else:
    114     saved_model_save.save(model, filepath, overwrite, include_optimizer,
--> 115                           signatures, options)
    116 
    117 

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/save.py in save(model, filepath, overwrite, include_optimizer, signatures, options)
     76     # we use the default replica context here.
     77     with distribution_strategy_context._get_default_replica_context():  # pylint: disable=protected-access
---> 78       save_lib.save(model, filepath, signatures, options)
     79 
     80   if not include_optimizer:

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/saved_model/save.py in save(obj, export_dir, signatures, options)
    897   # Note we run this twice since, while constructing the view the first time
    898   # there can be side effects of creating variables.
--> 899   _ = _SaveableView(checkpoint_graph_view)
    900   saveable_view = _SaveableView(checkpoint_graph_view)
    901 

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/saved_model/save.py in __init__(self, checkpoint_view)
    185           #  variables on first run.
    186           concrete_functions = (
--> 187               function._list_all_concrete_functions_for_serialization())  # pylint: disable=protected-access
    188         else:
    189           concrete_functions = [function]

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py in _list_all_concrete_functions_for_serialization(self)
    825     concrete_functions = []
    826     for args, kwargs in seen_signatures:
--> 827       concrete_functions.append(self.get_concrete_function(*args, **kwargs))
    828     return concrete_functions
    829 

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in get_concrete_function(self, *args, **kwargs)
    554   def get_concrete_function(self, *args, **kwargs):
    555     if not self.call_collection.tracing:
--> 556       self.call_collection.add_trace(*args, **kwargs)
    557     return super(LayerCall, self).get_concrete_function(*args, **kwargs)
    558 

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in add_trace(self, *args, **kwargs)
    426             fn.get_concrete_function(*args, **kwargs)
    427 
--> 428         trace_with_training(True)
    429         trace_with_training(False)
    430       else:

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in trace_with_training(value, fn)
    424           utils.set_training_arg(value, self._training_arg_index, args, kwargs)
    425           with K.learning_phase_scope(value):
--> 426             fn.get_concrete_function(*args, **kwargs)
    427 
    428         trace_with_training(True)

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in get_concrete_function(self, *args, **kwargs)
    555     if not self.call_collection.tracing:
    556       self.call_collection.add_trace(*args, **kwargs)
--> 557     return super(LayerCall, self).get_concrete_function(*args, **kwargs)
    558 
    559 

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py in get_concrete_function(self, *args, **kwargs)
    917       # In this case we have not created variables on the first call. So we can
    918       # run the first trace but we should fail if variables are created.
--> 919       concrete = self._stateful_fn.get_concrete_function(*args, **kwargs)
    920       if self._created_variables:
    921         raise ValueError("Creating variables on a non-first call to a function"

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py in get_concrete_function(self, *args, **kwargs)
   2432       args, kwargs = None, None
   2433     with self._lock:
-> 2434       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
   2435       if self.input_signature:
   2436         args = self.input_signature

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   2701 
   2702       self._function_cache.missed.add(call_context_key)
-> 2703       graph_function = self._create_graph_function(args, kwargs)
   2704       self._function_cache.primary[cache_key] = graph_function
   2705       return graph_function, args, kwargs

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2591             arg_names=arg_names,
   2592             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2593             capture_by_value=self._capture_by_value),
   2594         self._function_attributes,
   2595         # Tell the ConcreteFunction to clean up its graph once it goes out of

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    896       kwarg_shapes = None
    897     func_args = _get_defun_inputs_from_args(
--> 898         args, arg_names, flat_shapes=arg_shapes)
    899     func_kwargs = _get_defun_inputs_from_kwargs(
    900         kwargs, flat_shapes=kwarg_shapes)

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py in _get_defun_inputs_from_args(args, names, flat_shapes)
   1125   """Maps Python function positional args to graph-construction inputs."""
   1126   return _get_defun_inputs(
-> 1127       args, names, structure=args, flat_shapes=flat_shapes)
   1128 
   1129 

~/.pyenv/versions/sg/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py in _get_defun_inputs(args, names, structure, flat_shapes)
   1189     specified_names = [arg.name for arg in tensor_specs if arg.name]
   1190     if specified_names and len(specified_names) < len(tensor_specs):
-> 1191       raise ValueError("If specifying TensorSpec names for nested structures, "
   1192                        "either zero or all names have to be specified.")
   1193 

ValueError: If specifying TensorSpec names for nested structures, either zero or all names have to be specified.
Full output

GAT

tf.keras.models.save_model: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
tf.saved_model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)

GCN

tf.keras.models.save_model: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
tf.saved_model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)

PPNP

tf.keras.models.save_model: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
tf.saved_model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)

APPNP

tf.keras.models.save_model: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
tf.saved_model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)

GraphSAGE

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

HinSAGE

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

Number of clusters 1
0 cluster has size 2708

ClusterGCN

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

Attri2Vec

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

RGCN

tf.keras.models.save_model: AttributeError("'NoneType' object has no attribute 'replace'",)
tf.saved_model.save: AttributeError("'NoneType' object has no attribute 'replace'",)
model.save: AttributeError("'NoneType' object has no attribute 'replace'",)

ComplEx

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

DistMult

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

GCNSupervisedGraphClassification

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

DeepGraphInfomax

tf.keras.models.save_model: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
tf.saved_model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)
model.save: ValueError('If specifying TensorSpec names for nested structures, either zero or all names have to be specified.',)

DeepGraphInfomax

INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets
INFO:tensorflow:Assets written to: /tmp/assets

Expected behavior

Every model should support saving.

Environment

Operating system: Darwin-18.6.0-x86_64-i386-64bit
Python version:

3.6.9 (default, Jul 10 2019, 12:25:55) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)]

Package versions:

absl-py==0.8.0
ansiwrap==0.8.4
appdirs==1.4.3
appnope==0.1.0
astor==0.8.0
atomicwrites==1.3.0
attrs==19.3.0
backcall==0.1.0
black==19.10b0
bleach==3.1.0
boto==2.49.0
boto3==1.9.230
botocore==1.12.230
cachetools==4.0.0
certifi==2019.9.11
chardet==3.0.4
Click==7.0
coverage==4.5.4
coveralls==1.8.2
cycler==0.10.0
decorator==4.4.0
defusedxml==0.6.0
docopt==0.6.2
docutils==0.15.2
entrypoints==0.3
gast==0.2.2
gensim==3.8.0
google-auth==1.10.0
google-auth-oauthlib==0.4.1
google-pasta==0.1.7
gprof2dot==2019.11.30
grpcio==1.23.0
h5py==2.10.0
idna==2.8
importlib-metadata==0.23
ipykernel==5.1.3
ipython==7.9.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
isodate==0.6.0
jedi==0.15.1
Jinja2==2.10.3
jmespath==0.9.4
joblib==0.13.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.0.0
jupyter-core==4.6.1
Keras==2.2.5
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
llvmlite==0.30.0
Mako==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mistune==0.8.4
more-itertools==7.2.0
mplleaflet==0.0.5
mypy==0.750
mypy-extensions==0.4.3
nbclient==0.1.0
nbconvert==5.6.1
nbformat==4.4.0
networkx==2.3
notebook==6.0.2
numba==0.46.0
numpy==1.17.2
oauthlib==3.1.0
opt-einsum==3.1.0
packaging==19.2
pandas==0.25.1
pandocfilters==1.4.2
papermill==1.2.1
parso==0.5.1
pathspec==0.6.0
pdoc3==0.7.2
pexpect==4.7.0
pickleshare==0.7.5
pluggy==0.13.1
prometheus-client==0.7.1
prompt-toolkit==2.0.10
protobuf==3.9.1
ptyprocess==0.6.0
py==1.8.0
py-cpuinfo==5.0.0
py4j==0.10.7
pyasn1==0.4.8
pyasn1-modules==0.2.7
pydot==1.4.1
Pygments==2.4.2
Pympler==0.8
pyparsing==2.4.2
pyrsistent==0.15.6
pyspark==2.4.4
pyspark-stubs==2.4.0.post6
pytest==5.3.1
pytest-benchmark==3.2.2
pytest-cov==2.8.1
pytest-profiling==1.7.0
pytest-repeat==0.8.0
python-dateutil==2.8.0
pytz==2019.2
PyYAML==5.1.2
pyzmq==18.1.1
qtconsole==4.6.0
rdflib==4.2.2
regex==2019.12.9
requests==2.22.0
requests-oauthlib==1.3.0
rsa==4.0
s3transfer==0.2.1
scikit-learn==0.21.3
scipy==1.4.1
seaborn==0.10.0
Send2Trash==1.5.0
six==1.12.0
smart-open==1.8.4
-e git+git@github.com:stellargraph/stellargraph.git@c337c18fa391169234ccc0f46b699493d96748a7#egg=stellargraph
tenacity==6.0.0
tensorboard==2.1.1
tensorflow==2.1.0
tensorflow-estimator==2.1.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
textwrap3==0.9.2
toml==0.10.0
tornado==6.0.3
tqdm==4.42.1
traitlets==4.3.3
treon==0.1.3
typed-ast==1.4.0
typing-extensions==3.7.4.1
urllib3==1.25.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.6
widgetsnbextension==3.5.1
wrapt==1.11.2
zipp==0.6.0

Additional context

N/A
@huonw huonw added bug Something isn't working sg-library labels Apr 16, 2020
@huonw huonw mentioned this issue Apr 16, 2020
Closed
@huonw
Copy link
Member Author

huonw commented Apr 17, 2020

https://community.stellargraph.io/t/valueerror-if-specifying-tensorspec-names-for-nested-structures-either-zero-or-all-names-have-to-be-specified/97/3?u=huon notes that tf.compat.v1.keras.experimental.export_saved_model and tf.compat.v1.keras.experimental.load_from_saved_model work.

@jakubvedral
Copy link

jakubvedral commented May 29, 2020 •
edited

Hi,
For me in TensorFlow 2.2 event the workaround with
tf.compat.v1.keras.experimental.export_saved_model and tf.compat.v1.keras.experimental.load_from_saved_model does not work :(

Still the same error as you described above.

@huonw huonw self-assigned this Jun 9, 2020
@huonw
Copy link
Member Author

huonw commented Jun 10, 2020

I'm investigating this now.

Main finding so far: this seems to be specifically associated with sparse GraphConvolution, so a work-around for sufficiently small graphs is to pass sparse=True to the original FullBatchNodeGenerator.

@huonw
Copy link
Member Author

huonw commented Jun 10, 2020

This seems to be some problem with layers that take sparse tensors as input, even if they're not being used, e.g.:

Reduced form (via creduce and manual editing):

import tensorflow as tf

class SqueezedSparseConversion(tf.keras.layers.Layer):
    def call(self, inputs):
        return tf.SparseTensor([(0, 1)], [0.1], (3, 3))

class GraphConvolution(tf.keras.layers.Layer):
    def call(self, inputs):
        return inputs[0]

x_t = tf.keras.Input(0)
sp = SqueezedSparseConversion()(x_t)
out = GraphConvolution()([x_t, sp])

m = tf.keras.Model([x_t], out)
m.summary()
m.save("")
Full output 
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 0)]          0                                            
__________________________________________________________________________________________________
squeezed_sparse_conversion (Squ (3, 3)               0           input_1[0][0]                    
__________________________________________________________________________________________________
graph_convolution (GraphConvolu (None, 0)            0           input_1[0][0]                    
                                                                 squeezed_sparse_conversion[0][0] 
==================================================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0
__________________________________________________________________________________________________
2020-06-10 15:18:41.594860: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-06-10 15:18:41.608329: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbb46965a70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-10 15:18:41.608343: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-06-10 15:18:41.673648: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
Traceback (most recent call last):
  File "issue-1251.py", line 17, in <module>
    m.save("")
  File ".../site-packages/tensorflow/python/keras/engine/network.py", line 1052, in save
    signatures, options)
  File ".../site-packages/tensorflow/python/keras/saving/save.py", line 138, in save_model
    signatures, options)
  File ".../site-packages/tensorflow/python/keras/saving/saved_model/save.py", line 78, in save
    save_lib.save(model, filepath, signatures, options)
  File ".../site-packages/tensorflow/python/saved_model/save.py", line 951, in save
    obj, export_dir, signatures, options, meta_graph_def)
  File ".../site-packages/tensorflow/python/saved_model/save.py", line 1022, in _build_meta_graph
    _ = _SaveableView(checkpoint_graph_view)
  File ".../site-packages/tensorflow/python/saved_model/save.py", line 204, in __init__
    function._list_all_concrete_functions_for_serialization())  # pylint: disable=protected-access
  File ".../site-packages/tensorflow/python/eager/def_function.py", line 841, in _list_all_concrete_functions_for_serialization
    concrete_functions.append(self.get_concrete_function(*args, **kwargs))
  File ".../site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 546, in get_concrete_function
    self.call_collection.add_trace(*args, **kwargs)
  File ".../site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 421, in add_trace
    fn.get_concrete_function(*args, **kwargs)
  File ".../site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function
    return super(LayerCall, self).get_concrete_function(*args, **kwargs)
  File ".../site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
  File ".../site-packages/tensorflow/python/eager/def_function.py", line 877, in _get_concrete_function_garbage_collected
    *args, **kwargs)
  File ".../site-packages/tensorflow/python/eager/function.py", line 2496, in _get_concrete_function_garbage_collected
    graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
  File ".../site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File ".../site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File ".../site-packages/tensorflow/python/framework/func_graph.py", line 898, in func_graph_from_py_func
    args, arg_names, flat_shapes=arg_shapes)
  File ".../site-packages/tensorflow/python/framework/func_graph.py", line 1132, in _get_defun_inputs_from_args
    args, names, structure=args, flat_shapes=flat_shapes)
  File ".../site-packages/tensorflow/python/framework/func_graph.py", line 1196, in _get_defun_inputs
    raise ValueError("If specifying TensorSpec names for nested structures, "
ValueError: If specifying TensorSpec names for nested structures, either zero or all names have to be specified.
Package versions 
absl-py==0.7.1
alabaster==0.7.12
appnope==0.1.0
astor==0.8.0
astroid==2.3.3
astunparse==1.6.3
attrs==19.3.0
avro==1.8.2
avro-python3==1.8.2
awscli==1.16.198
Babel==2.8.0
backcall==0.1.0
bleach==3.1.0
boto==2.49.0
boto3==1.9.208
botocore==1.12.208
cachetools==4.1.0
certifi==2019.6.16
chardet==3.0.4
colorama==0.3.9
cycler==0.10.0
dataclasses==0.6
decorator==4.4.0
defusedxml==0.6.0
Deprecated==1.2.7
dgl==0.3
docutils==0.14
entrypoints==0.3
gast==0.3.3
gensim==3.8.0
google-auth==1.14.3
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.25.0
gviz-api==1.9.0
h5py==2.10.0
idna==2.8
imagesize==1.2.0
importlib-metadata==1.3.0
ipykernel==5.1.3
ipython==7.11.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.15.2
Jinja2==2.10.3
jmespath==0.9.4
joblib==0.13.2
json5==0.8.5
jsonschema==3.2.0
jupyter-client==5.3.4
jupyter-core==4.6.1
jupyterlab==1.2.4
jupyterlab-server==1.0.6
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.3
Mako==1.0.14
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mccabe==0.6.1
mistune==0.8.4
more-itertools==8.0.2
mpmath==1.1.0
nbconvert==5.6.1
nbformat==4.4.0
nbsphinx==0.6.1
networkx==2.3
notebook==6.0.2
numexpr==2.7.1
numpy==1.18.4
oauthlib==3.1.0
opt-einsum==3.2.1
packaging==20.3
pandas==1.0.3
pandocfilters==1.4.2
parso==0.5.2
patsy==0.5.1
pdoc3==0.6.3
pexpect==4.7.0
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==3.0.2
protobuf==3.9.0
ptyprocess==0.6.0
py-spy==0.3.2
py4j==0.10.7
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydantic==0.32.1
PyGithub==1.45
Pygments==2.5.2
PyJWT==1.7.1
pylint==2.4.4
Pympler==0.8
pyparsing==2.4.2
pyrsistent==0.15.6
pyspark==2.4.3
python-dateutil==2.8.0
python-igraph==0.8.0
pytz==2019.1
PyYAML==5.1
pyzmq==18.1.1
requests==2.22.0
requests-oauthlib==1.3.0
rsa==3.4.2
s3transfer==0.2.1
scikit-learn==0.21.3
scipy==1.4.1
Send2Trash==1.5.0
sgplatform==0.5.2a0
six==1.12.0
smart-open==1.8.4
snowballstemmer==2.0.0
Sphinx==3.0.3
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
statsmodels==0.10.2
tabulate==0.8.3
tensorboard==2.2.1
tensorboard-plugin-profile==2.2.0
tensorboard-plugin-wit==1.6.0.post3
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
texttable==1.6.2
torch==1.1.0.post2
tornado==6.0.3
traitlets==4.3.3
typed-ast==1.4.0
urllib3==1.25.3
virtualenv==16.7.9
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==0.15.5
wrapt==1.11.2
xarray==0.15.1
zipp==0.6.0

@huonw huonw mentioned this issue Jun 11, 2020
Closed
@huonw
Copy link
Member Author

huonw commented Jun 11, 2020

I filed tensorflow/tensorflow#40373. I think we can work around this by not doing the SqueezedSparseConversion as a layer but instead doing it within each GraphConvolution. This does require passing around the indices, data and shape of the sparse matrix independently. This may have some downsides:

  • messier code
  • maybe a breaking change (but could be done in a way that isn't breaking)
  • maybe a performance hit, if constructing the sparse tensor for every GraphConvolution layer is a significant cost

@huonw huonw mentioned this issue Jun 12, 2020
Merged
huonw added a commit that referenced this issue Jun 16, 2020
@huonw
Add tests for saving and loading every model (#1676)
498356e 
This PR tests saving and loading of a `tf.keras.Model` instance for every model
in StellarGraph.

It does so by adding a `test_utils.model_save_load` helper function that
serialises and deserialisers a given model using various APIs provided by
TensorFlow:

- saving:
  - `tf.keras.Model.save` https://www.tensorflow.org/api_docs/python/tf/keras/Model#save
  - `tf.keras.models.save_model` https://www.tensorflow.org/api_docs/python/tf/keras/models/save_model
  - `tf.saved_model.save` https://www.tensorflow.org/api_docs/python/tf/saved_model/save
- loading: `tf.keras.models.load_model`
  https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model (not
  `tf.saved_model.load`
  https://www.tensorflow.org/api_docs/python/tf/saved_model/load, because it
  doesn't create a Keras `Model` object)

Several of these new tests fail (they're included but marked as xfail), due to:

- #1251
- #1252
- #1680 (found in this PR)
- #1681 (found in this PR)

As those issues are fixed, the xfailing can be removed.
This was referenced Jun 16, 2020
Merged
Merged
huonw added a commit that referenced this issue Jun 17, 2020
@huonw
Add names to RGCN weights (#1677)
c67b612 
A Keras Layer with weights without names cannot be saved
(tensorflow/tensorflow#36962), because an exception is
thrown:

```
AttributeError: 'NoneType' object has no attribute 'replace'
```

This PR adds names to the `add_weight` calls within
`RelationalGraphConvolution`, which were the only ones within all of
StellarGraph missing the `name=...` parameter.

This allows non-sparse RGCN models to be saved, but sparse ones still hit #1251.

See: #1252
huonw added a commit that referenced this issue Jun 17, 2020
@huonw
Fix (non-sparse) saving and loading of APPNP models (#1682)
20270d5 
The `final_layer` access was left over from #1204.

Sparse APPNP models still hit #1251.

See: #1680
@huonw huonw mentioned this issue Jun 18, 2020
Closed
@huonw
Copy link
Member Author

huonw commented Jun 18, 2020 •
edited

I filed tensorflow/tensorflow#40373.

That issue was closed as a duplicate of tensorflow/tensorflow#38465.

It seems that this is now fixed in tf-nightly (tested in #1689). That is, one can work around this problem by installing a version of https://pypi.org/project/tf-nightly/ .

As such, I think I'll put this on the back burner for now because we have other high priority tasks; but, please let me know if it's particularly important to you (and upgrading to tf-nightly temporarily isn't feasible).

@huonw huonw mentioned this issue Jun 29, 2020
Closed
@huonw
Copy link
Member Author

huonw commented Jun 29, 2020

TensorFlow released 2.3.0-rc0 recently, which includes the fix for this issue. #1742 validates that (the equivalent of) pip install tensorflow==2.3.0-rc0 works.

@huonw huonw linked a pull request Jul 28, 2020 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@huonw huonw

Labels
bug Something isn't working sg-library
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Un-xfail full-batch saving tests fixed by TensorFlow 2.3 stellargraph/stellargraph
2 participants
@huonw @jakubvedral