Merge and update variable and params annotations from typeshed #4246

Avasam · 2024-02-28T05:17:03Z

Summary of changes

Merge and update (because they were not all correct or exact) variable and params annotations from typeshed. Missing public annotations are missing from typeshed too.
This PR purposefully avoids adding TypeVars, overloads and return types. Those will all be tackled in a follow-up PR. Return types are inferred at this step.

Step 3.1 of #2345 (comment) but only for pkg_resources

Pull Request Checklist

Changes have tests (type-checking tests are coming right after Fix all mypy issues #3979 )
News fragment added in newsfragments/.
(See documentation for details)

Avasam · 2024-02-29T03:18:27Z

pkg_resources/__init__.py

+_NestedStr = Union[str, Iterable[Union[str, Iterable["_NestedStr"]]]]
+_InstallerType = Callable[["Requirement"], Optional["Distribution"]]
+_PkgReqType = Union[str, "Requirement"]
+_EPDistType = Union["Distribution", _PkgReqType]
+_MetadataType = Optional["IResourceProvider"]


These aliases come from the typeshed stub. Please lmk if you'd prefer different names or to make them public.

pkg_resources/__init__.py

Avasam · 2024-03-07T23:49:05Z

pkg_resources/__init__.py

+    egg_name: str
+    egg_info: str
+


Suggested change

egg_name: str

egg_info: str

Actually there's no assurance that an egg is even found.

Avasam · 2024-04-21T20:19:46Z

As draft until #4262 and #4267 are completed

abravalheri · 2024-03-08T12:55:11Z

pkg_resources/__init__.py

+_MetadataType = Optional["IResourceProvider"]
+# Any object works, but let's indicate we expect something like a module (optionally has __loader__ or __file__)
+_ModuleLike = Union[object, types.ModuleType]
+_AdapterType = Callable[..., Any]  # Incomplete


Thank you very much Avasam for having a look on this.

I believe that instead of _AdapterType it is better to define something like _DistributionFinder, _NamespaceHandler, _ProviderFactory ...

Then _find_adapter can possibly be defined using typing.overload for the special cases (untested).

I plan on updating with:

_ProviderFactoryType = Callable[[_ModuleLike], "IResourceProvider"] _DistFinderType = Callable[[_T, str, bool], Iterable["Distribution"]] _NSHandlerType = Callable[[_T, str, str, types.ModuleType], Optional[str]] _AdapterT = TypeVar( "_AdapterT", _DistFinderType[Any], _ProviderFactoryType, _NSHandlerType[Any] ) # ... def _find_adapter( registry: Mapping[type, _AdapterT], ob: object, ) -> _AdapterT: ...

in a follow-up PR, but I can add it to this one if you'd like.

If you are planning that as a follow-up PR, that should be fine.

Thank you very much!

I've already created the follow-up here: 2c2874d (#4355)

pkg_resources/extern/__init__.py

abravalheri · 2024-05-09T18:46:15Z

pkg_resources/__init__.py

@@ -99,7 +123,20 @@
    stacklevel=2,
 )

+if TYPE_CHECKING:
+    from _typeshed import StrPath
+    from types import _LoaderProtocol


This is an import of a private member... Does _typeshed provide an equivalent? If not, we probably have to open an FR there before being able to import it here...

That comes from typeshed. https://github.com/python/typeshed/blob/main/stdlib/types.pyi#L321C1-L322C63
It's private to help prevent accidental runtime imports because typeshed didn't start using @type_check_only until very recently (and I believe we'll keep these names private anyway). In other words, _LoaderProtocol doesn't exist in the types module.

We could potentially ask to move _LoaderProtocol from types.pyi to typeshed/__init__.pyi and see what other typeshed maintainers think.

Alternatively, I can declare the protocol here close to the other aliases

If _LoaderProtocol is undocumented, I think we should not import it.

If typeshed maintainers are happy to make an utility definition of LoaderProtocol in _typeshed accessible for public usage that would be great. Otherwise if the protocol is small, declaring the protocol inline should also be fine.

(Brain dump: probably having an _typeshed.importlib module that helps to address the concerns in python/typeshed#11882 (comment) would be even greater).

I'm in favor of python/typeshed#11890 (comment)

abravalheri · 2024-05-09T18:54:11Z

pkg_resources/__init__.py

@@ -353,7 +390,7 @@ class UnknownExtra(ResolutionError):
    """Distribution doesn't have an "extra feature" of the given name"""


-_provider_factories = {}
+_provider_factories: Dict[Type[_ModuleLike], _AdapterType] = {}


Instead of _AdapterType, it would be more "sound" here to have a callable that, passed a *module* object, returns an ``IResourceProvider`` for that module., would it?

See comment https://github.com/pypa/setuptools/pull/4246/files#r1596038268
This would become

_provider_factories: Dict[Type[_ModuleLike], _ProviderFactoryType] = {} ```

abravalheri · 2024-05-09T18:55:24Z

pkg_resources/__init__.py

@@ -363,7 +400,9 @@ class UnknownExtra(ResolutionError):
 DEVELOP_DIST = -1


-def register_loader_type(loader_type, provider_factory):
+def register_loader_type(
+    loader_type: Type[_ModuleLike], provider_factory: _AdapterType


Same as previous: instead of _AdapterType, it would be more "sound" here to have a callable that, passed a *module* object, returns an ``IResourceProvider`` for that module., would it?

See comment https://github.com/pypa/setuptools/pull/4246/files#r1596038268
This would become

def register_loader_type( loader_type: Type[_ModuleLike], provider_factory: _ProviderFactoryType, ) -> None: ...

abravalheri · 2024-05-09T19:00:09Z

pkg_resources/__init__.py

@@ -2023,7 +2114,7 @@ def __init__(self, importer):
 ] = _declare_state('dict', '_distribution_finders', {})


-def register_finder(importer_type, distribution_finder):
+def register_finder(importer_type: type, distribution_finder: _AdapterType):


Instead of _AdapterType, it would be more "sound" here to have a callable that, passed a path item and the importer instance, yields ``Distribution`` instances found on that path item or something that matches the same type as the _distribution_finders variable above, would it?

See comment https://github.com/pypa/setuptools/pull/4246/files#r1596038268
This would become:

def register_finder( importer_type: Type[_T], distribution_finder: _DistFinderType[_T] ) -> None:

abravalheri · 2024-05-09T19:04:49Z

pkg_resources/__init__.py

@@ -2201,7 +2296,7 @@ def resolve_egg_link(path):
 )


-def register_namespace_handler(importer_type, namespace_handler):
+def register_namespace_handler(importer_type: type, namespace_handler: _AdapterType):


Instead of _AdapterType, it would be more "sound" here to have a callable like this:

def namespace_handler(importer, path_entry, moduleName, module): # return a path_entry to use for child packages

wouldn't it?

See comment https://github.com/pypa/setuptools/pull/4246/files#r1596038268
This would become:

def register_namespace_handler( importer_type: Type[_T], namespace_handler: _NSHandlerType[_T], ) -> None: ...

abravalheri · 2024-05-09T19:10:39Z

pkg_resources/__init__.py

@@ -3183,7 +3333,7 @@ def _always_object(classes):
    return classes


-def _find_adapter(registry, ob):
+def _find_adapter(registry: Mapping[type, _AdapterType], ob: object):


Here the way to go is probably have 3 definitions using typing.overload for the 3 different types of adapters and compatible ob (something like _ProviderFactory+ _LoaderProtocol; _DistributionFider+_Importer; _NamespaceHandler+_Importer, with _Importer a protocol that is compatible with PathEntryFinder and/or MetaPathFinder? ... considering the limitations in python/typeshed#11541 and python/typeshed#2468)

See comment https://github.com/pypa/setuptools/pull/4246/files#r1596038268

Avasam · 2024-05-18T21:21:32Z

@abravalheri in-code comments have been updated to reflect that fixes have been done upstream. I don't think we need to wait for the next mypy release. type-ignore can be removed when updating to mypy 1.11

abravalheri · 2024-05-22T09:30:58Z

Thank you very much.

Avasam force-pushed the pkg_resources-variables-params-annotations branch 5 times, most recently from c9c9332 to b0e1a1e Compare February 28, 2024 06:10

Avasam commented Feb 29, 2024

View reviewed changes

jaraco approved these changes Mar 3, 2024

View reviewed changes

This was referenced Mar 6, 2024

Make pkg_resources.find_nothing return an (empty) Generator of Distribution #4249

Closed

Type pkg_resources._declare_state and make it work statically #4258

Merged

pkg_resources: Runtime fixes to make typing annotations work #4262

Merged

Avasam commented Mar 6, 2024

View reviewed changes

pkg_resources/__init__.py Outdated Show resolved Hide resolved

Avasam mentioned this pull request Mar 6, 2024

Add type hints to setuptools #2345

Open

Avasam commented Mar 7, 2024

View reviewed changes

jaraco force-pushed the main branch from 9296481 to 6067fa4 Compare April 18, 2024 21:31

Avasam marked this pull request as draft April 21, 2024 20:13

Avasam requested a review from jaraco April 21, 2024 20:21

Avasam marked this pull request as ready for review May 7, 2024 19:12

Avasam force-pushed the pkg_resources-variables-params-annotations branch 4 times, most recently from 595bda1 to 092d919 Compare May 9, 2024 17:15

abravalheri reviewed May 9, 2024

View reviewed changes

Avasam mentioned this pull request May 12, 2024

Generic typing for register methods in pkg_resources #4355

Merged

2 tasks

Merge and update variable and params annotations from typeshed

adc40f2

Avasam force-pushed the pkg_resources-variables-params-annotations branch from ac1950e to adc40f2 Compare May 13, 2024 16:40

Avasam requested a review from abravalheri May 18, 2024 21:21

Merge branch 'main' into pkg_resources-variables-params-annotations

168d815

abravalheri merged commit 131b6e9 into pypa:main May 22, 2024
20 of 22 checks passed

Avasam deleted the pkg_resources-variables-params-annotations branch May 22, 2024 13:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge and update variable and params annotations from typeshed #4246

Merge and update variable and params annotations from typeshed #4246

Avasam commented Feb 28, 2024 •

edited

Avasam Feb 29, 2024 •

edited

Avasam Mar 7, 2024

Avasam commented Apr 21, 2024 •

edited

abravalheri Mar 8, 2024 •

edited

Avasam May 9, 2024 •

edited

abravalheri May 10, 2024 •

edited

Avasam May 13, 2024

abravalheri May 9, 2024

Avasam May 9, 2024 •

edited

abravalheri May 10, 2024

Avasam May 10, 2024

abravalheri May 9, 2024

Avasam May 9, 2024 •

edited

abravalheri May 9, 2024

Avasam May 9, 2024

abravalheri May 9, 2024

Avasam May 9, 2024

abravalheri May 9, 2024

Avasam May 9, 2024

abravalheri May 9, 2024

Avasam May 9, 2024

Avasam commented May 18, 2024

abravalheri commented May 22, 2024

Merge and update variable and params annotations from typeshed #4246

Merge and update variable and params annotations from typeshed #4246

Conversation

Avasam commented Feb 28, 2024 • edited

Summary of changes

Pull Request Checklist

Avasam Feb 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Avasam commented Apr 21, 2024 • edited

abravalheri Mar 8, 2024 • edited

Choose a reason for hiding this comment

Avasam May 9, 2024 • edited

Choose a reason for hiding this comment

abravalheri May 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Avasam May 9, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Avasam May 9, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Avasam commented May 18, 2024

abravalheri commented May 22, 2024

Avasam commented Feb 28, 2024 •

edited

Avasam Feb 29, 2024 •

edited

Avasam commented Apr 21, 2024 •

edited

abravalheri Mar 8, 2024 •

edited

Avasam May 9, 2024 •

edited

abravalheri May 10, 2024 •

edited

Avasam May 9, 2024 •

edited

Avasam May 9, 2024 •

edited