Support for AWS S3 paths as Pydantic Types? #4271
Replies: 2 comments 1 reply
-
Yes I think this could be added in V2. I think most of the features listed here should be possible to support with The most useful thing for us would be a list of valid and invalid S3 paths - perhaps from some existing libraries unit tests - if you can supply that, I'll make sure this is added to V2. More general point for dicussion - should URL validation be included in pydantic-core or kept in python? |
Beta Was this translation helpful? Give feedback.
-
did this go anywhere? |
Beta Was this translation helpful? Give feedback.
-
I work at Amazon and we are planning to use Pydantic for validating data. A lot of this data is stored on AWS S3 and we want to validate S3 paths.
Concretely:
AWS S3 paths are basically Urls. I have looked within Pydantic and they do not seem to be supported.
They follow a few rules, here are the important ones:
s3://
(this is meant to be like thehttps://
schema).s3://
is the bucket name, which must be between 3 and 64 chars and consist only of lowercase letters, numbers, dots (.), and hyphens (-) (See here for a few more restrictions). E.g.s3://my-bucket/
ands3://my-bucket
(without/
) both have the bucket namemy-bucket
.s3://my-bucket/some/key/path
, thesome/key/path
is the key. Notice it does not start with/
./
to create virtual "folders" (internally, the sharding is done based on key). A key is an S3 "folder" if ends with/
, e.g.s3://my-bucket/some/folder/path/
, otherwise it is an "object".It would be so nice to have a Pydantic Type like
S3Path
, which when passed a string in the constructor, does the following:/
)I have looked in the Pydantic docs and on Google, nothing seems to exist for this use case. I've tried to extend
AnyUrl
but it has not worked.Could this please be added?
Beta Was this translation helpful? Give feedback.
All reactions