Type hinting would generate a lot of import statements (possibly leading to circular imports). Use strings instead of imported names?

[vladiibine]

The need for the annotations to be valid Python expressions could generate a lot of otherwise unneeded (and ugly) import statements.

Two solutions seem appropriate on the first look:

1. The hinting could be done by using strings, if the referenced objects are in another module.

def foo(processor: 'myapp.processors.Processor',
        document: 'myapp.documents.Document')

1. Type comments could be used for type hinting, instead of actual function annotations

def foo(x,  # type: int
        y)  # type: float

Of course, the solutions I give are simple suggestions, I'm sure there is a better way to solve this problem.

Arguments against using actual imported names for type arguments:

1. It's ugly to have lots of extra imports, only used for hinting
2. The startup time of a lot of applications would become much slower - because importing lots of names from all over the place would require a lot more stuff to be loaded upfront.
3. Higher changes of circular imports - it's really not a problem one would expect to introduce. One could imagine people choosing not to specify type hints, out of fear of circular imports, or people introducing circular imports, by mistake, when creating "useful" type hints.

[gvanrossum]

The current rules for type hints in string literals or comments still require the imports. What makes you think you will need a lot of extra imports? Have you experienced this while trying to convert some code to use type hints? That would be very interesting feedback, I'd like to know more.

[vladiibine]

I've been using the kind of type hinting that Pycharm (and Sphinx I think) supports - :type myvar: SomeClass.
Although these hints are placed in comment strings, in order for them to "work", those symbols must be either proper imported names, fully qualified names, or relatively referenced names (with the relative import syntax).

The problems I had with the first approach (importing the names) are exactly the kind of problems this PEP would introduce.

So for the specific details: I'm working on a Django project (closed source), which has evolved over time, and includes about 50 models. We have methods on our models which accept as parameters instances of other models. In the current state of our project, to avoid circular references, we already use a pattern allowed by Django, whereby we specify the references (foreign key relationships) between these models as strings.

Now admittedly, many of the circular import problems could be solved by refactoring the models that reference each other, to live in the same module, but the effort for this would at this point would be non trivial and would require rethinking the architecture of the entire project.

This seems a little too much work to enjoy the simple benefits of type hinting.

To show however that not only the project I'm working with will have this problem, I cloned a popular django repository, namely django-cms.

I try to show how in django, the intrinsic relationship Model -> Manager -> QuerySet -> Model is so well spread that at least for the django community, the type hinting, as proposed, will cause a lot of chaos.

https://github.com/vladiibine/django-cms/tree/type-hinting-issue-105

In the first commit, I show how Django was already preparing for cyclic dependency problems, and solved this issue in exactly the way I'm proposing - using strings instead of names
vladiibine/django-cms@553982f

In the second commit, I again break the application, merely by importin some names, and in the third commit, I fix the app by moving the classes around (I move the PageQuerySet in the same module as the model Page). However acceptable this would seem, as far as I understand Django, what I've done is not OK. Also, please notice that I had to fit more classes inside a module JUST to be able to document some function return types.
vladiibine/django-cms@4e1a1b0

In the 4th commit I again try to specify types, and get to a functioning application state. This state unfortunally is quite sad, because I can not document function parameters with their proper types. That is, in Django, at least a part of a Manager's methods are expected to return QuerySets. It's quite unfortunate to have to yet again move the Manager class in the same module as the Model and the QuerySet class. Of course, we can imagine what happens if model methods take parameters or return instances of other models: All the Models, Managers and QuerySets will end up in 1 file, which would be crazy.
vladiibine/django-cms@426c23c

Please note that DjangoCMS is not a model-rich application. I counted ~15 models here. As I mentioned the project I'm working on has around 50 models, which are more tightly connected. While I understand that this wouldn't be an example of Python project architecture, I'm concerned that projects like this (or ones even more coupled) would not be able to benefit much from the type hinting. These projects would however be in most need of it, to make them more readable, understandable and easier to refactor.

I mentioned that the loading time of applications would be affected, but have not yet tried to investigate this, though I'm hoping to sell that argument together with this one: If type hinting requires names to be imported, the coupling of modules will become more problematic.

[gvanrossum]

I see. This is indeed nothing to sneeze at. A style that limits imports to modules (instead of classes) would have much less of a problem here, because at the module level circular imports are not much of a problem, and then using strings for annotations could preserve the "must be imported" requirement. But this is clearly not the style that most Django projects use, so it would be kind of a hollow recommendation. Stubs could probably bypass the problem (the imports would still be there but a stub does not need to follow the strict rules of define-before-use when it comes to type hints). But then you lose the type checking of your model code.

The reason why we require the imports for type hints (even in strings or comments) is that otherwise the type checker would have to guess about the full name of a class referenced in a type hint -- many class names are not unique and even if they are the search space is unwieldy. "Fully-qualified names" might be considered an exception but there is still the ambiguity if a top-level module name matches a locally-defined name.

[JukkaL]

I agree that type annotations introducing circular module dependencies can be a pain. I've also seen that in the mypy codebase. Refactoring can help with this, but it can result in huge modules with dozens of classes which is not great for maintainability.

We had a related discussion (though without the Django context) before: #34

Just supporting a non-imported class name in annotation isn't really practical, since we don't know where to look for the class definition, as Guido pointed out.

Currently the only reasonable approach that I've been able to come up with (and this is what I currently use in mypy) is to import a module without from ... import and use a fully-qualified string literal type. Example:

import app.model   # introduces circular dependency

def foo(t: 'app.model.Thing') -> 'app.model.Thing': ...

This is ugly and verbose, and it introduces circular dependencies, but it has worked for me without needing much refactoring.

Another idea (that I mentioned in the above issue) is to introduce a new primitive for introducing references to a class defined in another module, without the need to import anything. For example:

from typing import TypeAlias

Thing = TypeAlias('app.model.Thing')  # must be fully qualified name?

def foo(t: Thing) -> Thing: ...

This doesn't introduce a circular module dependency and the annotation is less ugly. However, the type alias definition is pretty verbose. Maybe we can also support relative module references (e.g., TypeAlias('.model.Thing')). A TypeAlias is basically a wrapper around str and doesn't have any interesting logic. It could define some methods such as .get() to actually find the target type but that's not important.

To make this more practical, we can define a module that only contains a bunch of type aliases (note that it doesn't import any model modules):

# app/types.py
from typing import TypeAlias

Thing = TypeAlias('app.model.Thing')
Manager = TypeAlias('app.manager.Manager')
... # etc. for other model classes

Now we can import the type aliases from this module and code looks cleaner. We just need to add an import -- however, now we can use from import without introducing a circular dependency since types doesn't import any other app modules:

from app.types import Thing

def foo(t : Thing) -> Thing: ...

The tradeoff is that we may need to add a definition to types.py for every new model object we define. I think that this is not a terrible burden, but it's not very nice compared to Java, for example.

If we'd have the freedom to modify the language, we could add a new kind of import that defines a (type) alias instead of importing a name. For example (straw man syntax proposal):

from app.model import alias Thing

# now Thing is an instance _Alias('app.model.Thing') and we didn't import app.model --
# there's no circular dependecy

def foo(t: Thing) -> Thing: ...

The above code would be equivalent to this code (I'm assuming that _Alias gets added to the builtins, but it can live anywhere):

Thing = _Alias('app.model.Thing')

def foo(t: Thing) -> Thing: ...

_Alias would be similar to TypeAlias above, but it wouldn't be specific to typing and could be used by Django, for example, as an alternative to string literal escaping. The problem with this is that mistyping an imported name wouldn't be an error, since we can't look into the target module when evaluating import alias. However, this is no worse than string literal escapes, and a type checker or linter would verify that the imported name is actually defined.

Another issue is that using an alias as a regular value outside a type annotation would probably result in an AttributeError or TypeError, which may be a little confusing. Having useful exception error messages would probably go a long way towards making this a non-issue.

[NYKevin]

I have some code which has an issue similar to this, though my type annotations are not (exactly) compatible with mypy's syntax. I solved it at runtime by basically building strings out of __name__ and calling str.split('.')/'.'.join(), as you can see here, since I don't want to repeat the full name of the module (that would defeat the purpose of relative imports). It's rather ugly and I doubt it can be parsed statically, but it does leave the annotations in a reasonable state for Sphinx and other tools which actually import my modules. Because the other module also imports this one, and uses it for non-type-hinting things, I feel it would be difficult to make this work correctly with circular imports. I suppose I could force the import order using __init__.py, but I dislike that kind of kludgery.

[gvanrossum]

I've written up a brief discussion of this issue with a suggestion how to go about it in the PEP: https://hg.python.org/peps/rev/06fbe54fcfe1

[164747]

I bump into this issue from time to time. Often the circular import statement only exists due to type hinting and are not used by python. In the (very) long term future, would the syntax

from my_module himport A

be a good solution? That is a statement ignored by python but read by the type-hinter as a normal import.

[gvanrossum]

On Thu, Dec 3, 2015 at 1:02 PM, 164747 notifications@github.com wrote:

I bump into this issue from time to time. Often the circular import
statement only exists due to type hinting and are not used by python. In
the (very) long term future, would the syntax

from my_module himport A

be a good solution? That is a statement ignored by python but read by the
type-hinter as a normal import.

I don't know about that particular syntax. Personally I would rather
experiment with these:

# type: import my_module
# type: from my_module import A

since they can be made to work in all versions of Python, today.

[NYKevin]

Personally, I like Guido's comment syntax better than inventing a new keyword (which at this point would be 3.6+ only at the absolute earliest). I mean, I suppose you could restrict the new keyword to stub files, but... eww.

[164747]

I wouldn't assume my syntax to be better than Guido's =)

[vladiibine]

I'm not sure if this is a good idea, but if python had some special comments where only metadata lives, this is how I imagine metadata could look like.

@# meta info about the code goes here
@# and then some more meta
@# return: mymodule.A
@# param x: othermodule.B
def foo(x):
  return something

This wouldn't mess up with the interpreter at run-time, but could be used to give static hints about stuff.

I'm sorry if I made Python look too much like something else. I also realize this isn't necessarily very compatible with the python 3.5 type hinting.

[o11c]

Python really does need a generic "this is metadata" syntax.

Decorators can sometimes be used for functions/classes, and annotations can be used for function arguments/return, but even that is horribly limited (e.g. the need for forward declarations using strings, which fails to work in some obscure set of cases that I can't remember).

Other cases that exist in the wild are: if 0: blocks (which only works for code that parses as valid Python, unlike #if 0 blocks in C which only need to be lexically valid), string literals and comments (which nobody can agree to the format of).

To misquote the Zen of Python: there should be five, and preferably only five, obvious ways to do it.

---

In order to support older python versions, a codec can be used. In order to implement an efficient, incremental, codec, it should operate a simple lexer (minimal awareness of string/comments is mandatory, more is unnecessary). There should probably be two translation modes: the default which strips everything out, and another that produces a more complete (though not executable) AST.

The following two characters are completely unused in Python: $ and ?. Additionally, </code> is unused in Python3 and the codec _could_ change its meaning in python2, but it interferes with markdown. !is only valid as part of the!=token, and` is only valid at end of line. There are plenty of two-character combinations however (such as the @# just suggested while I was writing this - though that is only usable at statement scope, since it collides with matmul at expression scope).

I've been tossing around ideas, but haven't found anything I'm happy with.

Some difficult problems:

• It is absolutely essential to preserve both line and column information for the post-encoding form. This places a lot of constraints, since an indentation-based language can't just substitute deleted content with a space. This means that statement-or-higher scope metadata will have to be appended, not prepended or wrapped - but this looks really weird at expression scope.
• We need something general enough that it can be applied to nearly any AST node (or sequence of AST nodes). In particular, expressions (both lvalue and rvalue), statements, blocks, subblocks (since if 1: is a wart), empty lines, and names in other places (such as function arguments).
• There needs to be some sort of unavoidable namespacing at the start of the metadata, but also allow structured arguments. For example, it should be easy for some coverage metadata to say "disable coverage on these lines if on python3.2 or on python2"
• We need to actually ship some way of dealing accessing the extended ASTs, as well as involving all the other people who have written parsers. This is directly at odds with the "need to be able to apply metadata to any AST node" goal.

[gvanrossum]

Please take the more general discussion elsewhere (e.g. python-ideas).