Typing support

[pp-mo]

I reckon that this line is our first use of type-hints in Iris code (:+1: 🚀 @jamesp ).
This is consistent with dropping support for Python3.6

What do we feel about this, is there a case for adding type hints to major sections of the API ?
Perhaps we should also make a statement in the 3.2 whatsnew

[trexfeathers]

I would imagine adding as I go, in the same way as I do with Numpy docstrings (which contradicts #3840 but we just haven't found time yet).

[jamesp]

@pp-mo see also #4272. Some great input from Jon Seddon

[jamesp]

Also, while type hints were not "formalised" in some sense until relatively recently, I think that annotations and the typing library preceed python 3.6. Adding type annotations should not cause issues with running on 3.6 if that were necessary, they are not interpreted at runtime.

Edit: Terrible grammar

[bjlittle]

Just had a lovely typing conversation with @rcomer regarding #4294, where being a very good iris citizen, she added type annotations to her PR.

However, this conversation raised quite a lot of open questions that really do need a group steer to make the process of adding typing in iris way, way less painful i.e., IMHO we really do need to start to curate an evolving typing best practice

1. Bootstrapping typing stub files seems like the scalable and maintainable way to go. iris and it's data model is complex, so it would be ideal to start simple with common iris typing axioms and build/scale outwards
2. Dealing with arrays is just confusing (due to a lack of typing maturity atm even with numpy.typing?). We need clarity on generic typing and when to discriminate between specific real (numpy), lazy (dask), ndarray and masked cases
3. When is it best to specialise with specific array dtype and shape?
4. What audience is typing targeting? Users? Developers? or both? i.e., this leads to the question of type hinting private classes, functions and methods
5. We need clarity on when to use collections.abc.Sequence vs collections.abc.Iterable vs collections.abc.Iterator vs typing.Iterable
6. We need to know the pros and cons of when to use a typing alias vs a typing.TypeVar
7. Typing the code is only half of the problem. How the typing is rendered in documentation is the other. It's very easy to create a typing documentation abomination 😱
8. Typing can render documentation and doc-strings hard / very difficult to read. Somehow xarray appear to have somewhat squared that circle, see https://xarray.pydata.org/en/v0.13.0/generated/xarray.merge.html (thanks to @jamesp for sharing this observation)

Just thought I'd openly capture and share our conversation questions here... as I can only assume it might resonate for anyone else who has dipped their toe into the typing world for bloated large legacy code bases.

Have I missed or misrepresented anything @rcomer ?

Ideally, it would be great to agree on a way forward using #4294 as a concrete use case and roll from there...

[rcomer]

Thanks @bjlittle, that's a great summary. I think the only other thing on my list was the rather specific question of how to hint for something that could be a built-in or numpy type (but basically the same thing). E.g. where the code clearly expects that the variable may be an int or np.integer is it enough to just hint that it's an int?

[rcomer]

Typing changes in #4294 are currently contained in the last 4 commits.

[bjlittle]

@jamesp also shared this link, which is a great read, see https://sethmlarson.dev/blog/2021-10-18/tests-arent-enough-case-study-after-adding-types-to-urllib3

[rcomer]

For the rendered docs, it looks like xarray have simply turned off the type hints:
https://github.com/pydata/xarray/blob/e056cacdca55cc9d9118c830ca622ea965ebcdef/doc/conf.py#L112
https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints

The advantages of typing discussed at #4272 seem to focus on how editors can interpret them, rather than documentation. I'm particularly not keen on how Optional is shown as the typing Optional is specifically not the same thing as an optional argument, which seems unintuitive to me. Would much rather it said "or None"

So, if we're voting, I think I'll vote for turning them off in the docs.

[trexfeathers]

I can see the logic to this. Although it does of course mean that we need to remember to keep including the types in the docstring as well as the signature (not needing to do this was something @pp-mo was looking forward to!).

[pp-mo]

keep including the types in the docstring as well as the signature

It's a shame for DRY (and while trialling typechecks elsewhere, I've already had problems with the two getting out-of-step)
But I totally agree that docs readability has to come first.

[rcomer]

PR to turn them off here: #4510

[pp-mo]

I think I may have found a "better way" for this using autodoc_typehints = "description"
See comment here

[pp-mo]

I think some of the other negative behaviours noted above may have eased, too.

E.G.

@jamesp not keen on how Optional is shown ... Would much rather it said "or None"

I believe that is now fixed : napoleon now renders Optional(int) as int | None.

[rcomer]

We need clarity on when to use collections.abc.Sequence vs collections.abc.Iterable vs collections.abc.Iterator vs typing.Iterable

Today I learned that if you specify thing is an Iterable, PyRight complains if you do len(thing). So in this case Sequence appears to be the way to go.

[Edit: assuming PyRight is actually Right...]

[rcomer]

Although a set is an Iterable and has a length, but is not a Sequence...

[jamesp]

Maybe Sized is what you're looking for?

https://docs.python.org/3/library/typing.html#typing.Sized

[rcomer]

Hmmm. I want to check the length and loop through it...

[rcomer]

Maybe I want a Collection
https://docs.python.org/3/library/typing.html#typing.Collection

[rcomer]

👀 handy table here

https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes

[trexfeathers]

Closing following the creation of #5924 - we do want to do this one way or another