You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello.
My initial data is about 3.8 gigabyte. I have used different batch_sizes but again, I have this serialization size issue.
I am using a Tesla T4, 15 GB for that, but I face the same issue when I use CPU as well.
This is the complete error: Traceback (most recent call last): File "/home/gkarant/mobispaces/tensorflow_federated/tff.py", line 198, in <module> main() File "/home/gkarant/mobispaces/tensorflow_federated/tff.py", line 183, in main updated_server_state = training_process.next(server_state, federated_train_data) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/computation/computation_impl.py", line 148, in __call__ return self._context_stack.current.invoke(self, arg) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/sync_execution_context.py", line 65, in invoke return self._async_runner.run_coro_and_return_result( File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/async_utils.py", line 149, in run_coro_and_return_result return future.result() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/retrying.py", line 119, in retry_coro_fn raise e File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/retrying.py", line 109, in retry_coro_fn result = await fn(*args, **kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 236, in invoke arg = await tracing.wrap_coroutine_in_current_trace_context( File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 406, in _wrapped return await coro File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 120, in _ingest ingested = await asyncio.gather(*ingested) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 127, in _ingest return await executor.create_value(val, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 207, in async_trace result = await fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/remote_executor.py", line 236, in create_value value_proto, type_spec = serialize_value() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/remote_executor.py", line 229, in serialize_value return value_serialization.serialize_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 374, in serialize_value return _serialize_federated_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 321, in _serialize_federated_value federated_value_proto, it_type = serialize_value(v, type_spec.member) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 370, in serialize_value return _serialize_sequence_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 272, in _serialize_sequence_value value_proto.sequence.serialized_graph_def = _serialize_dataset(value) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 166, in _serialize_dataset raise ValueError( ValueError: Serialized size of Dataset (459161045 bytes) exceeds maximum allowed (104857600 bytes)
Is there a dedicated solution for that?
Thank you
The text was updated successfully, but these errors were encountered:
Hello.
My initial data is about 3.8 gigabyte. I have used different batch_sizes but again, I have this serialization size issue.
I am using a Tesla T4, 15 GB for that, but I face the same issue when I use CPU as well.
This is the complete error:
Traceback (most recent call last): File "/home/gkarant/mobispaces/tensorflow_federated/tff.py", line 198, in <module> main() File "/home/gkarant/mobispaces/tensorflow_federated/tff.py", line 183, in main updated_server_state = training_process.next(server_state, federated_train_data) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/computation/computation_impl.py", line 148, in __call__ return self._context_stack.current.invoke(self, arg) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/sync_execution_context.py", line 65, in invoke return self._async_runner.run_coro_and_return_result( File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/async_utils.py", line 149, in run_coro_and_return_result return future.result() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/retrying.py", line 119, in retry_coro_fn raise e File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/retrying.py", line 109, in retry_coro_fn result = await fn(*args, **kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 236, in invoke arg = await tracing.wrap_coroutine_in_current_trace_context( File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 406, in _wrapped return await coro File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 120, in _ingest ingested = await asyncio.gather(*ingested) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 127, in _ingest return await executor.create_value(val, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 207, in async_trace result = await fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/remote_executor.py", line 236, in create_value value_proto, type_spec = serialize_value() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/remote_executor.py", line 229, in serialize_value return value_serialization.serialize_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 374, in serialize_value return _serialize_federated_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 321, in _serialize_federated_value federated_value_proto, it_type = serialize_value(v, type_spec.member) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 370, in serialize_value return _serialize_sequence_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 272, in _serialize_sequence_value value_proto.sequence.serialized_graph_def = _serialize_dataset(value) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 166, in _serialize_dataset raise ValueError( ValueError: Serialized size of Dataset (459161045 bytes) exceeds maximum allowed (104857600 bytes)
Is there a dedicated solution for that?
Thank you
The text was updated successfully, but these errors were encountered: