Marshmallow Validation Error

[lihao881230]

Hello, we are attempting to run the scripts from the extract_from_npx module. When running the following command: python -W ignore -m ecephys_spike_sorting.modules.extract_from_npx --input_json <C:\<path\to\input json> --output_json C:\<path to output json>, we run into the following error:

Traceback (most recent call last): File "C:\Users\lihao\anaconda3\envs\npx\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "C:\Users\lihao\anaconda3\envs\npx\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\ecephys_spike_sorting\modules\extract_from_npx\__main__.py", line 71, in <module> main() File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\ecephys_spike_sorting\modules\extract_from_npx\__main__.py", line 59, in main output_schema_type=OutputParameters) File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\argschema\argschema_parser.py", line 175, in __init__ result = self.load_schema_with_defaults(self.schema, args) File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\argschema\argschema_parser.py", line 274, in load_schema_with_defaults result = utils.load(schema, args) File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\argschema\utils.py", line 418, in load results = schema.load(d) File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\marshmallow\schema.py", line 707, in load postprocess=True, File "C:\Users\lihao\anaconda3\envs\npx\lib\site-packages\marshmallow\schema.py", line 867, in _do_load raise exc marshmallow.exceptions.ValidationError: {'directories': {'kilosort_output_tmp': ['Unknown field.']}, 'quality_metrics_params': ['Unknown field.'], 'median_subtraction_params': ['Unknown field.'], 'ks_postprocessing_params': ['Unknown field.'], 'waveform_metrics': ['Unknown field.'], 'mean_waveform_params': ['Unknown field.'], 'depth_estimation_params': ['Unknown field.'], 'noise_waveform_params': ['Unknown field.'], 'kilosort_helper_params': ['Unknown field.'], 'ephys_params': ['Unknown field.']}

We are not sure how to proceed, as this seems to be an error in parsing the dictionaries. We are currently using argschema==2.0.2, and we are using a new anaconda environment for this as well. Please let us know if you have any suggestions, thanks!

[jsiegle]

This is an issue with argschema v2+...by default, it doesn't allow you to include input fields that aren't explicitly listed in the schema. The solution is to either edit your input JSON files so they contains fields for only one module at a time, or to use pip to downgrade to the pinned versions of argschema (1.17.5) and marshmallow (2.19.2).

Let me know if that works!

[lihao881230]

Yes that seemed to resolve the validation error issue, thanks!

[Selmaan]

I had the same problem trying to run the 'batch processing' script with only 'quality metrics' module uncommented, and downgrading fixed it for me as well

[vncntprvst]

I get this issue as well, despite using the appropriate versions of argschema and marshmallow:

Traceback (most recent call last):
  File "v:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "v:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "V:\Code\SpikeSorting\ecephys_spike_sorting\ecephys_spike_sorting\modules\extract_from_npx\__main__.py", line 71, in <module>
    main()
  File "V:\Code\SpikeSorting\ecephys_spike_sorting\ecephys_spike_sorting\modules\extract_from_npx\__main__.py", line 59, in main
    output_schema_type=OutputParameters)
  File "V:\Code\SpikeSorting\ecephys_spike_sorting\.venv\lib\site-packages\argschema\argschema_parser.py", line 175, in __init__
    result = self.load_schema_with_defaults(self.schema, args)
  File "V:\Code\SpikeSorting\ecephys_spike_sorting\.venv\lib\site-packages\argschema\argschema_parser.py", line 274, in load_schema_with_defaults
    result = utils.load(schema, args)
  File "V:\Code\SpikeSorting\ecephys_spike_sorting\.venv\lib\site-packages\argschema\utils.py", line 422, in load
    raise mm.ValidationError(errors)
marshmallow.exceptions.ValidationError: {'extract_from_npx_params': {'settings_xml': ['Field may not be null.'], 'npx_directory': ['Field may not be null.']}, 'common_files': {'settings_json': ['Field may not be null.'], 'probe_json': ['Field may not be null.']}}

I suspect a package conflict between Anaconda and pipenv, but I can't find what goes wrong.

> pip show marshmallow
Name: marshmallow
Version: 2.19.2
Summary: A lightweight library for converting complex datatypes to and from native Python datatypes.
Home-page: https://github.com/marshmallow-code/marshmallow
Author: Steven Loria
Author-email: sloria1@gmail.com
License: MIT
Location: v:\code\spikesorting\ecephys_spike_sorting\.venv\lib\site-packages
Requires:
Required-by: argschema

> pip show argschema  
Name: argschema
Version: 1.17.5
Summary: a wrapper for setting up modules that can have parameters specified by command line arguments,       json_files, or dictionary objects. Providing a common wrapper for data processing modules.
Home-page: https://github.com/AllenInstitute/argschema
Author: Forrest Collman,David Feng
Author-email: forrestc@alleninstitute.org
License: UNKNOWN
Location: v:\code\spikesorting\ecephys_spike_sorting\.venv\lib\site-packages
Requires: marshmallow, numpy
Required-by: ecephys-spike-sorting

Any idea?
Thanks

[vncntprvst]

After installing on Linux I get the same error, so it's not due to a conflict.
I'll open a separate issue.