Overriding issue for optional dataclass argument

[function2-llx]

🐛 Bug report

When using an optional dataclass, fields specified first will be wiped by later fields specification.

To reproduce

from dataclasses import dataclass

from jsonargparse import ArgumentParser

@dataclass
class A:
    x: int
    y: int

def main():
    parser = ArgumentParser()
    parser.add_argument('--a', type=A | None)
    args = parser.parse_args([
        '--a.x', '1',
        '--a.y', '2',
    ])
    print(args)

if __name__ == '__main__':
    main()

Expected behavior

Should output as if the dataclass parameter is required:

Namespace(a=Namespace(x=1, y=2))

Actual Result

error: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

Run with JSONARGPARSE_DEBUG=1:

2023-11-21 18:37:44,314 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(a=None)
2023-11-21 18:37:44,357 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(x=None, y=None)
2023-11-21 18:37:44,360 - ArgumentParser - DEBUG - Parsed command line arguments: ['--x=1']
2023-11-21 18:37:44,361 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(x=None, y=None)
2023-11-21 18:37:44,363 - ArgumentParser - DEBUG - Parsed command line arguments: ['--y=2']
2023-11-21 18:37:44,364 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(x=None, y=None)
2023-11-21 18:37:44,364 - ArgumentParser - DEBUG - Parsed object: Namespace(x=None, y=2)
2023-11-21 18:37:44,366 - ArgumentParser - ERROR - Validation failed: Key "x" is required but not included in config object or its value is None.
2023-11-21 18:37:44,366 - ArgumentParser - ERROR - Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)
2023-11-21 18:37:44,366 - ArgumentParser - DEBUG - Debug enabled, thus raising exception instead of exit.
Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 1038, in check_required
    raise TypeError
TypeError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 1079, in check_config
    check_required(cfg, self)
  File ".../jsonargparse/_core.py", line 1040, in check_required
    raise TypeError(
TypeError: Key "x" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 441, in parse_object
    parsed_cfg = self._parse_common(
                 ^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 319, in _parse_common
    self.check_config(cfg, skip_required=skip_required)
  File ".../jsonargparse/_core.py", line 1087, in check_config
    raise type(ex)(message) from ex
TypeError: Validation failed: Key "x" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_typehints.py", line 695, in adapt_typehints
    vals.append(adapt_typehints(val, subtypehint, **adapt_kwargs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_typehints.py", line 840, in adapt_typehints
    val = parser.parse_object(val, defaults=sub_defaults.get() or list_item)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_deprecated.py", line 123, in patched_parse
    cfg = parse_method(*args, _skip_check=_skip_check, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 451, in parse_object
    self.error(str(ex), ex)
  File ".../jsonargparse/_core.py", line 1000, in error
    raise argument_error(message) from ex
argparse.ArgumentError: Validation failed: Key "x" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_typehints.py", line 498, in _check_type
    raise ex
  File ".../jsonargparse/_typehints.py", line 485, in _check_type
    val = adapt_typehints(val, self._typehint, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_typehints.py", line 700, in adapt_typehints
    raise_union_unexpected_value(typehint, val, vals)
  File ".../jsonargparse/_typehints.py", line 590, in raise_union_unexpected_value
    raise ValueError(
ValueError: Does not validate against any of the Union subtypes
Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
Errors:
  - Validation failed: Key "x" is required but not included in config object or its value is None.
  - Expected a <class 'NoneType'>
Given value type: <class 'jsonargparse._namespace.Namespace'>
Given value: Namespace(x=None, y=2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 1081, in check_config
    check_values(cfg)
  File ".../jsonargparse/_core.py", line 1071, in check_values
    raise ex
  File ".../jsonargparse/_core.py", line 1066, in check_values
    self._check_value_key(action, val, key, ccfg)
  File ".../jsonargparse/_core.py", line 1351, in _check_value_key
    value = action._check_type(value, cfg=cfg)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_typehints.py", line 508, in _check_type
    raise TypeError(f'Parser key "{self.dest}"{elem}:\n{error}') from ex
TypeError: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 394, in parse_args
    parsed_cfg = self._parse_common(
                 ^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 319, in _parse_common
    self.check_config(cfg, skip_required=skip_required)
  File ".../jsonargparse/_core.py", line 1087, in check_config
    raise type(ex)(message) from ex
TypeError: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./test.py", line 20, in <module>
    main()
  File "./test.py", line 13, in main
    args = parser.parse_args([
           ^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_deprecated.py", line 123, in patched_parse
    cfg = parse_method(*args, _skip_check=_skip_check, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 403, in parse_args
    self.error(str(ex), ex)
  File ".../jsonargparse/_core.py", line 1003, in error
    raise argument_error(message) from ex
argparse.ArgumentError: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

Environment

• jsonargparse version: 4.27.0
• Python version: 3.11.6
• How jsonargparse was installed: pip install jsonargparse[signatures, omegaconf]
• OS: Linux (Ubuntu 22.04)

[mauvilsa]

Thank you for reporting! Just to give an update. I did look at this and it is not a trivial fix. I do have a potential idea, but haven't had time for it.

[eunwoosh]

are there any updates here?

[mauvilsa]

are there any updates here?

No, still the same as my last comment.