You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It just fails with error: mlflow.exceptions.MlflowException: The model that is attempting to be saved has been loaded into memory with an incompatible configuration. If you are using the accelerate library to load your model, please ensure that it is saved only after loading with the default device mapping. Do not specify device_map and please try again.
But if i just remove device map it works. But without device_map its hard to traning model since its not follow distrubted traning.
So anyone have an idea how fix this. Even in code its mentioned cooment as and has not been updated.
# Verify that the model has not been loaded to distributed memory
# NB: transformers does not correctly save a model whose weights have been loaded
# using accelerate iff the model weights have been loaded using a device_map that is
# heterogeneous. There is a distinct possibility for a partial write to occur, causing an
# invalid state of the model's weights in this scenario. Hence, we raise.
# We might be able to remove this check once this PR is merged to transformers:
# https://github.com/huggingface/transformers/issues/20072
if _is_model_distributed_in_memory(built_pipeline.model):
raise MlflowException(
"The model that is attempting to be saved has been loaded into memory "
"with an incompatible configuration. If you are using the accelerate "
"library to load your model, please ensure that it is saved only after "
"loading with the default device mapping. Do not specify `device_map` "
"and please try again."
)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello,
I have seen an issue, where if i mention
When trying to log model with
It just fails with error:
mlflow.exceptions.MlflowException: The model that is attempting to be saved has been loaded into memory with an incompatible configuration. If you are using the accelerate library to load your model, please ensure that it is saved only after loading with the default device mapping. Do not specify
device_mapand please try again.
But if i just remove device map it works. But without device_map its hard to traning model since its not follow distrubted traning.
So anyone have an idea how fix this. Even in code its mentioned cooment as and has not been updated.
Beta Was this translation helpful? Give feedback.
All reactions