Version mismatch between library and pretrained model?

[expectopatronum]

Describe the bug

I installed the most recent nussl version (1.1.3) using pip today and tried to run the Deep Clustering example, which first downloads the model dpcl-wsj2mix-model.pth.

This is the error message:

Traceback (most recent call last):                                                                                                                                                                                 
  File "explore_nussl_embeddings.py", line 25, in <module>                                                                                                                                                         
    separator = nussl.separation.deep.DeepClustering(                                                                                                                                                              
  File "/home/verena/miniconda3/lib/python3.8/site-packages/nussl/separation/deep/deep_clustering.py", line 36, in __init__                                                                                        
    self.load_model(model_path, device=device)                                                                                                                                                                     
  File "/home/verena/miniconda3/lib/python3.8/site-packages/nussl/separation/base/deep_mixin.py", line 35, in load_model                                                                                           
    model_dict = safe_loader.load(model_path, 'cpu')                                                                                                                                                               
  File "/home/verena/miniconda3/lib/python3.8/site-packages/nussl/core/migration.py", line 83, in load                                                                                                             
    metadata = self._get_moved(metadata, model_dict, model_path)                                                                                                                                                   
  File "/home/verena/miniconda3/lib/python3.8/site-packages/nussl/core/migration.py", line 101, in _get_moved                                                                                                      
    raise SeparationException(f"Failed loading model. Expected to find "                                                                                                                                           
nussl.separation.base.separation_base.SeparationException: Failed loading model. Expected to find 'nussl_version' in /home/verena/.nussl/models/dpcl-wsj2mix-model.pth.

Steps To Reproduce

Running this example

Expected behavior
I expected the example above to work with the most current version of nussl.

What did happen
The example produced the above mentioned error when using nussl 1.1.3, but when using an older version (e.g. 1.0) it works.

Software versions

>>> import platform; print(platform.platform())
Linux-5.4.0-52-generic-x86_64-with-glibc2.10
>>> import sys; print("Python", sys.version)
Python 3.8.3 (default, May 19 2020, 18:47:26) 
[GCC 7.3.0]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.19.1
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.5.2
>>> import nussl; print("nussl", nussl.__version__)
nussl 1.1.3

[ethman]

Whoops! Sorry Verena. Thanks for catching that. We're in the process of adding more metadata to the models so that we can future-proof them. All of the code should be ready to go, but the models haven't been updated yet. So we added some strict checks to make sure that we're not loading models we don't recognize, which is what you hit. The models currently available for download are kind of old.

Here's a work around for the time being:

1. Download the model.
2. Load it in torch, outside of nussl (https://pytorch.org/tutorials/beginner/saving_loading_models.html)
3. Add a nussl_version key to the top-level dict of the loaded model. The value shouldn't matter but you can add 1.0.0.
4. Save it in torch.
5. Load the modified model in nussl.

Going forward, you should not need to do this. All models trained from 1.1.3 onward should be fully-future proof, with migration happening automatically on load in newer versions of nussl.

[andrezg98]

Hi! @ethman I have reproduced your indicated steps as follows:

model_path = nussl.efz_utils.download_trained_model('mask-inference-wsj2mix-model-v1.pth')

# workaround for known issue in the library
# https://github.com/nussl/nussl/issues/192#issuecomment-720689506
model = torch.load("/root/.nussl/models/mask-inference-wsj2mix-model-v1.pth", map_location=torch.device('cpu'))
model["nussl_version"] = "1.1.9"
torch.save(model, "/root/.nussl/models/mask-inference-wsj2mix-model-v1.pth")
##

audio_signal = nussl_custom_bach10[0]["mix"]
separator = nussl.separation.deep.DeepMaskEstimation(audio_signal, mask_type='soft', model_path=model_path)
estimates = separator()

and now I run in to this exception:

---

SeparationException Traceback (most recent call last)

in ()
24
25 audio_signal = nussl_custom_bach10[0]["mix"]
---> 26 separator = nussl.separation.deep.DeepMaskEstimation(audio_signal, mask_type='soft', model_path=model_path)
27 estimates = separator()
28

5 frames

/usr/local/lib/python3.7/dist-packages/nussl/separation/deep/deep_mask_estimation.py in init(self, input_audio_signal, model_path, device, **kwargs)
25 super().init(input_audio_signal, **kwargs)
26 if model_path is not None:
---> 27 self.load_model(model_path, device=device)
28 self.model_output = None
29 # audio channel dimension in a mask estimation model

/usr/local/lib/python3.7/dist-packages/nussl/separation/base/deep_mixin.py in load_model(self, model_path, device)
35
36 self.device = device
---> 37 model, metadata = SeparationModel.load(model_path)
38 model = model.to(device).eval()
39 self.model = model

/usr/local/lib/python3.7/dist-packages/nussl/ml/networks/separation_model.py in load(location)
234
235 safe_loader = SafeModelLoader()
--> 236 model_dict = safe_loader.load(location, 'cpu')
237 metadata = model_dict['metadata']
238

/usr/local/lib/python3.7/dist-packages/nussl/core/migration.py in load(self, model_path, device, expected_eval)
83 metadata = self._get_moved(metadata, model_dict, model_path)
84 self.eval = expected_eval
---> 85 metadata = self._validate_and_populate(metadata)
86 metadata['config'] = json.dumps(metadata['config'])
87 model_dict['metadata'] = metadata

/usr/local/lib/python3.7/dist-packages/nussl/core/migration.py in validate_and_populate(self, received)
165 for sub_key, type in expected_val.items():
166 sub_val = next_dict.get(sub_key, None)
--> 167 sub_result[sub_key] = self.load_types(type, sub_key, sub_val)
168 result[key] = sub_result
169

/usr/local/lib/python3.7/dist-packages/nussl/core/migration.py in _load_types(expected_type, key, val)
142 if val is not None:
143 if type(val) != expected_type:
--> 144 raise SeparationException(f'Expected type {expected_type} '
145 f'for key {key} but got {type(val)}!')
146 return val

SeparationException: Expected type None for key seed but got <class 'int'>!

[windspirit95]

Hi! @ethman I have reproduced your indicated steps as follows:

model_path = nussl.efz_utils.download_trained_model('mask-inference-wsj2mix-model-v1.pth')

# workaround for known issue in the library
# https://github.com/nussl/nussl/issues/192#issuecomment-720689506
model = torch.load("/root/.nussl/models/mask-inference-wsj2mix-model-v1.pth", map_location=torch.device('cpu'))
model["nussl_version"] = "1.1.9"
torch.save(model, "/root/.nussl/models/mask-inference-wsj2mix-model-v1.pth")
##

audio_signal = nussl_custom_bach10[0]["mix"]
separator = nussl.separation.deep.DeepMaskEstimation(audio_signal, mask_type='soft', model_path=model_path)
estimates = separator()

and now I run in to this exception:

SeparationException Traceback (most recent call last)
in ()
24
25 audio_signal = nussl_custom_bach10[0]["mix"]
---> 26 separator = nussl.separation.deep.DeepMaskEstimation(audio_signal, mask_type='soft', model_path=model_path)
27 estimates = separator()
28
5 frames
/usr/local/lib/python3.7/dist-packages/nussl/separation/deep/deep_mask_estimation.py in init(self, input_audio_signal, model_path, device, **kwargs)
25 super().init(input_audio_signal, **kwargs)
26 if model_path is not None:
---> 27 self.load_model(model_path, device=device)
28 self.model_output = None
29 # audio channel dimension in a mask estimation model
/usr/local/lib/python3.7/dist-packages/nussl/separation/base/deep_mixin.py in load_model(self, model_path, device)
35
36 self.device = device
---> 37 model, metadata = SeparationModel.load(model_path)
38 model = model.to(device).eval()
39 self.model = model
/usr/local/lib/python3.7/dist-packages/nussl/ml/networks/separation_model.py in load(location)
234
235 safe_loader = SafeModelLoader()
--> 236 model_dict = safe_loader.load(location, 'cpu')
237 metadata = model_dict['metadata']
238
/usr/local/lib/python3.7/dist-packages/nussl/core/migration.py in load(self, model_path, device, expected_eval)
83 metadata = self._get_moved(metadata, model_dict, model_path)
84 self.eval = expected_eval
---> 85 metadata = self._validate_and_populate(metadata)
86 metadata['config'] = json.dumps(metadata['config'])
87 model_dict['metadata'] = metadata
/usr/local/lib/python3.7/dist-packages/nussl/core/migration.py in validate_and_populate(self, received)
165 for sub_key, type in expected_val.items():
166 sub_val = next_dict.get(sub_key, None)
--> 167 sub_result[sub_key] = self.load_types(type, sub_key, sub_val)
168 result[key] = sub_result
169
/usr/local/lib/python3.7/dist-packages/nussl/core/migration.py in _load_types(expected_type, key, val)
142 if val is not None:
143 if type(val) != expected_type:
--> 144 raise SeparationException(f'Expected type {expected_type} '
145 f'for key {key} but got {type(val)}!')
146 return val
SeparationException: Expected type None for key seed but got <class 'int'>!

Hi,
I have run into the same issue with you, and I go to /usr/local/lib/python3.7/dist-packages/nussl/core/migration.py, change the type of key "seed" in line 48 to int, and it works.
Hope it helps you too!