Replies: 2 comments 8 replies
-
Is it not possible to use Hydra with pydantic without hydra-zen? When I use a pydantic type in a pydantic dataclass with Hydra I get a
If it is possible I would appreciate an example. |
Beta Was this translation helpful? Give feedback.
8 replies
-
Update: https://mit-ll-responsible-ai.github.io/hydra-zen/how_to/pydantic_guide.html |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi all!
This isn't a request or a question; it is really a show-and-tell 😄.
I am the lead dev on hydra-zen, which aims to make Hydra much easier to use and to enable novel capabilities in the framework. I believe that some of hydra-zen's function make it so that pydantic and Hydra can be used together! I am hoping to get some feedback from folks on the pydantic team.
Some Context About hydra-zen
The "crown jewel" of
hydra-zen
is the functionhydra_zen.builds
, which automatically generates structured configs (i.e. dataclasses that Hydra understands):Thus
builds
has the ability to inspect the signature of the target-object that we want to configure, and to create a Hydra-compatible dataclass that configures it. It also has many bells-and-whistles that keep users from snagging on any of Hydra's sharp edges. Most relevant to this discussion:builds
will automatically widen type-annotations that are not supported by Hydra's type-validation system.Using pydantic models in Hydra
So, how does
builds
provide a bridge between pydantic and Hydra? Let's check out an example:We can then call our Hydra-based application from the command line, and get the benefit of our pydantic model's validation:
Thus we've gained all the benefits of using the Hydra framework while also leveraging pydantic's type coercion and validation abilities!
How does this work?
Let's understand what is going on here;
builds(PydConf, populate_full_signature=True)
creates a dataclass whose signature matches that ofPydConf
, but whose type-annotations have been widened, where necessary, to appease Hydra's limited type-validation system. Thusval: PositiveFloat
->val: Any
. This is fine though, because pydantic will ultimately be performing type-validation viaPydConf
;HydraConf
is just a temporary "vessel" that defines the outer-most interface to our Hydra application/CLI.Within our task-function, we use Hydra's
instantiate
API to instantiate all of the objects that we have described with our configs. E.g.instantiate(HydraConf, val=1.0)
ends up callingPydConf(val=1.0)
and returns the result. Thus we ultimately instantiate the actual pydantic model before proceeding with the rest of our application.There's a (temporary) catch...
The Hydra-framework attempts to catch and re-raise errors with modified error messages, however
pydantic.ValidationError
has an atypical signature, and so Hydra's attempt to callpydantic.pydantic.ValidationError(new_err_msg)
produces aTypeError
. Thus the validation error raised in the above example gets obscured.Fortunately Hydra has merged a fix for this for their next minor release. (I also wonder if it might be worth refactoring
ValidationError
to accommodate this common usage.)One other way to use pydantic with Hydra
hydra_zen.builds
provides a generic ability to auto-apply (one or more) wrappers to any target object being configured/instantiated. Thuspydantic.validate_arguments
can be auto-applied to any/all interfaces being configured in your application:Feedback
I'd love to get your thoughts on the above.
Thanks for taking the time to read through this!
Beta Was this translation helpful? Give feedback.
All reactions