Broken `Type Hints` in PyTorch 0.4.0, related to IDEs(eq. PyCharm) #7318

kimdwkimdw · 2018-05-05T10:57:50Z

If you have a question or would like help and support, please ask at our
forums.

If you are submitting a feature request, please preface the title with [feature request].
If you are submitting a bug report, please fill in the following details.

Issue description

Recently, I figured out PyCharm cannot make auto-complete for torch.zeros.

PyCharm says

Cannot find reference 'zeros' in '__init__.py'

I dig it for a while I found broken Type Hints.

From these changes,
30ec06c
https://github.com/pytorch/pytorch/wiki/Breaking-Changes-from-Variable-and-Tensor-merge

Especially, 30ec06c#diff-14258fce7c17ccb97b488e64373b0803R308 @colesbury
This line cannot make Type Hints for lots of IDEs.

Originally, torch.zeros was in torch/_C/__init__.py
But, it moved to torch/_C/_VariableFunctions

Code example

https://gist.github.com/kimdwkimdw/50c18b5cf72c69c2d01bb4146c8a2b5c
This is Proof of Concept for this bug.

If you look at main.py

import T_B as torch

torch.p2()  # IDE can detect `p2`
torch.p1    # IDE cannot detect `p1`

System Info

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch or Caffe2:
How you installed PyTorch (conda, pip, source):
Any case for conda, pip, source.
Build command you used (if compiling from source):
OS: Any
PyTorch version: 0.4.0
Python version: 3.6.5
CUDA/cuDNN version: .
GPU models and configuration: .
GCC version (if compiling from source): .
CMake version: .
Versions of any other relevant libraries:.

The text was updated successfully, but these errors were encountered:

kimdwkimdw · 2018-05-05T11:00:31Z

any related point? #4568

nimcho · 2018-05-07T17:30:59Z

The same problem, pycharm is basically useless when using pytorch0.4. Almost everything is underlined either because "cannot find reference" or "is not callable".

sherif7810 · 2018-05-08T18:12:22Z

There's no auto-completion, quick definition & quick documentation for torch.tensor, torch.max and loss.backward.

OS: Linux (64-bit Fedora 27 with Gnome).
PyTorch version: 0.4.0.
How you installed PyTorch (conda, pip, source): pip3.
Python version: 3.6.5.
I use CPU (No CUDA).

zou3519 · 2018-05-08T18:53:20Z

If any of you have suggestions for how to fix this, please let us know!

It sounds like PyCharm's searching for functions could be improved, but I'm not sure what can be done from the PyTorch side. Using torch.tensor as an example, help(torch.tensor) gets the documentation while dir(torch) shows "tensor" as a member.

kimdwkimdw · 2018-05-09T05:33:50Z

@zou3519
I thought that below is not good code style.

for name in dir(_C._VariableFunctions):
    globals()[name] = getattr(_C._VariableFunctions, name)

from 30ec06c

globals()[name] is not Pythonic way. It is not only problem for PyCharm, but also for Python code style.

I thought _VariableFunctions is not need to be Class.

kimdwkimdw · 2018-05-10T16:26:08Z

I'm trying to make PR for this issue. Would you review? @zou3519

zou3519 · 2018-05-10T16:30:53Z

@kimdwkimdw I don't know what's going on with the global() there or what would make it better, but yes, please submit a PR and I will look at it :)

mattrobin · 2018-05-13T03:18:45Z

It sounds like PyCharm's searching for functions could be improved, but I'm not sure what can be done from the PyTorch side.

Just wanted to note that I have never run into this issue with any other package in PyCharm. That's not to say PyCharm shouldn't be doing things differently, but this issue does seem to be an uncommon case.

kimdwkimdw · 2018-05-16T02:07:49Z

I'm mainly working on these files.

tools/autograd/gen_autograd.py
tools/autograd/templates/python_torch_functions.cpp

torch/lib/include/torch/csrc/Module.cpp

Trying to figure out what's changed when generates C extensions.
I'll make PR soon.

JACKHAHA363 · 2018-06-02T01:45:42Z

Any updates?

kimdwkimdw · 2018-06-04T08:18:51Z

I made a working example in my fork.

If anyone want to use auto-complete first, try below.

1. git clone

git clone -b pytorch-interface https://github.com/kimdwkimdw/pytorch.git

2. Install PyTorch

https://github.com/kimdwkimdw/pytorch/tree/pytorch-interface#install-pytorch

3. Clear cache in PyCharm

usually caches are located in /Users/USERNAME/Library/Caches/PyCharm*/python_stubs/
check your project interpreter in PyCharm

Before pull request, I need to add more commits to organize results.

541435721 · 2018-06-04T13:44:57Z

install failed following the description above.
is there any simple way to install?

JC-S · 2018-06-19T16:35:16Z

I'm having the same problem. I've always loved pytorch and pycharm. Such a shame they don't work along well : (
Looking forward to updates.

JavisPeng · 2018-06-22T00:41:08Z

This is really strange when I used the same code in the python console with pycharm, it is normal, but in the editor. O__O "…

zou3519 · 2018-06-22T02:01:14Z

Going to bump the priority on this because a lot of users have been requesting this... we'll try our best to look into it more.

kimdwkimdw · 2018-06-23T09:04:03Z

A lot of users are requesting.... I should make a pull request this weekend.

kimdwkimdw · 2018-06-25T08:38:50Z

I merged my code with current master branch.

Type Hints are alive now.

MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install

Checkout PR - #8845

cc. @zou3519

541435721 · 2018-06-26T14:04:48Z

@kimdwkimdw should I reinstall pytorch from source? Is there any easy way to renew in the windows platform?

kimdwkimdw · 2018-06-26T16:20:31Z

@541435721
Yes, you should. Check out https://github.com/pytorch/pytorch#install-pytorch this page.
I don't know there is more simple way to build it.

nlgranger · 2018-06-27T14:47:43Z

It doesn't seem to change anything for Pycharm (2018.1, with rebuilt caches).

zou3519 · 2018-06-27T15:10:46Z

You should check out @kimdwkimdw 's pull request, not pytorch master. The pull request has not been merged into pytorch master yet.

This is the PR: #8845.

nlgranger · 2018-06-27T15:19:54Z

Yes, this what I did:

git remote add kimdwkimdw https://github.com/kimdwkimdw/pytorch.git
git fetch kimdwkimdw
git checkout --track kimdwkimdw/interface-0
git submodule update --init
CC=gcc-5 CXX=g++-5 CFLAGS="-march=native -O2" CXXFLAGS="-march=native -O2" python setup.py build
python setup.py install  --optimize=1 --skip-build

kimdwkimdw · 2018-06-27T16:40:01Z

@nlgranger you should delete python stubs manually. it cannot be deleted with 'invalidate cache'.

When you setup project, Pycharm initialize its python stub files.

After you delete python stub directory, restart Pycharm.

PyCharm will re-generate its python stub.

kleb0nas12 · 2019-01-12T21:06:02Z

the same here. Tried PyCharm , VS code , Spyder .. torch.cat impossible to get , the same as the Longtensor.

mattrobin · 2019-01-14T23:55:46Z

Throughout the comments in this issue, this has continually been billed as a problem related to PyCharm, and it's been mentioned that it's a problem with how PyCharm's internals and PyTorch's internals work together. This is a mischaracterization of the problem, and viewing it as such probably won't lead to the correct solution. PyCharm is taking a fairly standard approach to resolving packages, and, as has been mentioned by several others, this problem is not unique PyCharm. PyCharm just happens to be the most popular editor that runs into this issue with PyTorch. Any editor which does not dynamically resolve the package (for speed purposes) will have this problem. I think the fact that IPython/Jupyter/etc can resolve it correctly is the exception rather than the rule. I just wanted to emphasize this so that not too much focus is put on "making it work with PyCharm" rather than "making it work in general".

That being said, the stubs being worked on in this PR are a good "standard" solution, and should probably be able to resolve the issue in general. For those of you looking for the quick temporary solution, copy that stub file into your dist-packages/site-packages as explained in the gist. Manually add any other missing parts, and report those missing parts to ensure they get handled.

This is more a basis for discussion that a ready solution, as it does lots of funny things, and for many of them a better solution will be found. - Initial stab at creating a type torch/__init__.pyi . - We only do this for Python 3 because we want to use type hint introspection. - So far, we only aim at doing this for torch functions and torch.Tensor. - We need to import the newly build torch. Thus we do this at the end of the build. We use os.fork to only import the module in a child process. - We use an annotate decorator to be able to put type hints on the native python functions in a way that a) they're available in the usual place for Python 3 b) we stay Python 2 compatible - Some annotatons in torch/functional.py are provided as examples, but the remaining ones still need to be done. This could end up fixing pytorch#7318 This was backported from pytorch#12500 but with tests removed and the stub file checked in directly, so we don't have to also make sure all build shenanigans work. Master will be properly tested. The stub file was generated by running 'python setup.py create_pyi' and then copying the generated pyi file in the build directory (find -name *.pyi) to torch/

* create type hint stub files for module torch This is more a basis for discussion that a ready solution, as it does lots of funny things, and for many of them a better solution will be found. - Initial stab at creating a type torch/__init__.pyi . - We only do this for Python 3 because we want to use type hint introspection. - So far, we only aim at doing this for torch functions and torch.Tensor. - We need to import the newly build torch. Thus we do this at the end of the build. We use os.fork to only import the module in a child process. - We use an annotate decorator to be able to put type hints on the native python functions in a way that a) they're available in the usual place for Python 3 b) we stay Python 2 compatible - Some annotatons in torch/functional.py are provided as examples, but the remaining ones still need to be done. This could end up fixing #7318 This was backported from #12500 but with tests removed and the stub file checked in directly, so we don't have to also make sure all build shenanigans work. Master will be properly tested. The stub file was generated by running 'python setup.py create_pyi' and then copying the generated pyi file in the build directory (find -name *.pyi) to torch/ * Ignore pyi in flake8 Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Real fix for lint error Signed-off-by: Edward Z. Yang <ezyang@fb.com>

ezyang · 2019-01-30T20:37:03Z

Hi everyone! @t-vi's patch has merged to master, so if you update you should get working autocomplete for torch on master. Additionally, we committed a hand-generated type stub for 1.0.1, so when that release happens, autocomplete will also work for people on that release.

The type stub is probably not good enough for actually typechecking your code with mypy. We're tracking this follow up work in #16574, please pipe up if it affects you there. Also, we only fixed autocomplete on torch; if you are encountering problems with autocomplete on other modules (or you think there are missing identifiers on torch), please let us know with a bug report.

Thanks for your patience!

soumith · 2019-02-08T19:27:48Z

just fyi, the new v1.0.1 release now has this solved, and it's shipped.

TheCodez · 2019-02-09T22:42:02Z

@ezyang torch.cuda is missing in the pyi, for example when using torch.cuda.is_available.

AleksandrAlbert · 2019-02-12T02:01:57Z

I am getting intellisense for VSCode to partially work on torch 1.0.1, but some functions are still missing, eg, torch.as_tensor

vpj · 2019-02-12T05:14:21Z

I was also getting errors and no autocomplete for some functions in PyCharm (it was missing when I needed it most), and I don't think this is completely solved yet.

I wrote my first PyTorch program last weekend so not sure if I made some mistake with the setup.

soumith · 2019-02-12T05:17:06Z

@vpj it was fixed in PyTorch v1.0.1. Anything lower still has the issue. check print(torch.__version__)

vpj · 2019-02-12T05:45:37Z

Version is 1.0.1post2

Some definitions are missing in the init.pyi file.

PyCharm also seems to be causing some problems because it fails to infer return types. For instance, it doesn't seem to infer that the return value of torch.exp is a Tensor (however the return type is inferred with Tensor.exp). Although the return type is defined in the torch.exp method definition.

zhezh · 2019-02-12T06:00:15Z

yes, I also upgrade to '1.0.1.post2'
and auto completing not seems to work

soumith · 2019-02-12T06:05:32Z

so, apparently a few namespaces are still missing, though we fixed the larger problem.

I opened a new tracking task for this here: #16996

yarcowang · 2019-08-23T04:49:51Z

I'm not quite sure about this one? (Is this fixed?) Console works fine.
(But from torch.utils.data import DataLoader, Dataset works)

ezyang · 2019-08-23T19:41:03Z

@yarcowang We've been steadily adding more and more namespaces. This should work quite well on 1.2.0. If there are still missing things please file bugs.

yarcowang · 2019-08-24T02:17:14Z

@ezyang OK. I see. It seems a long-term bug.

PIPIKAI-Sung · 2020-01-21T03:49:21Z

I am a newer for pytorch. Thanks a lot, it works well for me.

akuma527 · 2020-02-25T07:12:25Z

Hello everyone, went through the entire thread. It's quite difficult to find whether the issue is resolved or not. I'm using torch1.4+cpu and torch.tensor still has warning issues. Can anyone help? Thanks

kmecpp · 2020-03-03T18:20:24Z

Also still experiencing this issue with torch.Tensor

ezyang · 2020-03-04T22:32:20Z

Thanks for the comments. We've been fixing lots of small type hint bugs over the last few months; if you notice things that are not working on the latest release or master please open bugs for them and we'll look. Thanks!

tjysdsg · 2020-03-12T10:54:36Z

Broken things that I have noticed in version 1.4.0:

torch.cuda.manual_seed # manual_seed not hinting
torch.cuda.manual_seed_all # manual_seed_all not hinting
torch.utils # cannot find utils
torch.backends # cannot find backends
torch.optim.lr_scheduler._LRScheduler.step() # argument "epoch" unfilled, while it's optional
torch.Tensor(4, 1) # unexpected arguments & argument "requires_grad" unfilled
torch.optim.Adadelta # cannot find Adadelta
torch.nn.TransformerEncoderLayer # cannot find TransformerEncoderLayer
torch.nn.TransformerEncoder # cannot find TransformerEncoder

zou3519 · 2020-03-12T14:42:36Z

@tjysdsg Thanks for reporting. Could you open a new issue with those cases for more visibility?

tjysdsg · 2020-03-13T01:29:39Z

@tjysdsg Thanks for reporting. Could you open a new issue with those cases for more visibility?

@zou3519 Here it is #34699, I'll also update the issue if I find anything new

kimdwkimdw changed the title ~~Broken Type Hints in PyTorch 0.4.0~~ Broken Type Hints in PyTorch 0.4.0, related to IDEs(eq. PyCharm) May 5, 2018

zou3519 added the todo Not as important as medium or high priority tasks, but we will work on these. label May 14, 2018

ssnl mentioned this issue Jun 18, 2018

[feature request] Make from_numpy method externally visible #8611

Closed

zou3519 added medium priority and removed todo Not as important as medium or high priority tasks, but we will work on these. labels Jun 22, 2018

kimdwkimdw added a commit to kimdwkimdw/pytorch that referenced this issue Jun 25, 2018

Add missing Type Hints to torch. issue pytorch#7318

6cebcdb

kimdwkimdw mentioned this issue Jun 25, 2018

Add missing Type Hints to torch. issue #7318 #8845

Closed

t-vi mentioned this issue Jan 11, 2019

MyPy Type Hinting Stub Files? #15839

Closed

ezyang mentioned this issue Jan 16, 2019

create type hint stub files for module torch #16089

Merged

ezyang closed this as completed Jan 30, 2019

soumith mentioned this issue Feb 12, 2019

namespaces torch.cuda, torch.device missing from pyi #16996

Closed

tjysdsg mentioned this issue Mar 13, 2020

Broken Hinting for PyCharm, torch 1.4.0 #34699

Closed

Broken Type Hints in PyTorch 0.4.0, related to IDEs(eq. PyCharm) #7318

Broken Type Hints in PyTorch 0.4.0, related to IDEs(eq. PyCharm) #7318

Comments

kimdwkimdw commented May 5, 2018 • edited

Issue description

Code example

System Info

kimdwkimdw commented May 5, 2018

nimcho commented May 7, 2018

sherif7810 commented May 8, 2018

zou3519 commented May 8, 2018

kimdwkimdw commented May 9, 2018 • edited

kimdwkimdw commented May 10, 2018

zou3519 commented May 10, 2018

mattrobin commented May 13, 2018

kimdwkimdw commented May 16, 2018

JACKHAHA363 commented Jun 2, 2018

kimdwkimdw commented Jun 4, 2018 • edited

1. git clone

2. Install PyTorch

3. Clear cache in PyCharm

541435721 commented Jun 4, 2018

JC-S commented Jun 19, 2018

JavisPeng commented Jun 22, 2018

zou3519 commented Jun 22, 2018

kimdwkimdw commented Jun 23, 2018

kimdwkimdw commented Jun 25, 2018 • edited

541435721 commented Jun 26, 2018

kimdwkimdw commented Jun 26, 2018

nlgranger commented Jun 27, 2018

zou3519 commented Jun 27, 2018

nlgranger commented Jun 27, 2018 • edited

kimdwkimdw commented Jun 27, 2018

kleb0nas12 commented Jan 12, 2019

mattrobin commented Jan 14, 2019 • edited

ezyang commented Jan 30, 2019

soumith commented Feb 8, 2019 • edited

TheCodez commented Feb 9, 2019

AleksandrAlbert commented Feb 12, 2019 • edited

vpj commented Feb 12, 2019

soumith commented Feb 12, 2019

vpj commented Feb 12, 2019

zhezh commented Feb 12, 2019

soumith commented Feb 12, 2019

yarcowang commented Aug 23, 2019 • edited

ezyang commented Aug 23, 2019

yarcowang commented Aug 24, 2019

PIPIKAI-Sung commented Jan 21, 2020

akuma527 commented Feb 25, 2020

kmecpp commented Mar 3, 2020 • edited

ezyang commented Mar 4, 2020

tjysdsg commented Mar 12, 2020 • edited

zou3519 commented Mar 12, 2020

tjysdsg commented Mar 13, 2020 • edited

Broken `Type Hints` in PyTorch 0.4.0, related to IDEs(eq. PyCharm) #7318

Broken `Type Hints` in PyTorch 0.4.0, related to IDEs(eq. PyCharm) #7318

kimdwkimdw commented May 5, 2018 •

edited

kimdwkimdw commented May 9, 2018 •

edited

kimdwkimdw commented Jun 4, 2018 •

edited

kimdwkimdw commented Jun 25, 2018 •

edited

nlgranger commented Jun 27, 2018 •

edited

mattrobin commented Jan 14, 2019 •

edited

soumith commented Feb 8, 2019 •

edited

AleksandrAlbert commented Feb 12, 2019 •

edited

yarcowang commented Aug 23, 2019 •

edited

kmecpp commented Mar 3, 2020 •

edited

tjysdsg commented Mar 12, 2020 •

edited

tjysdsg commented Mar 13, 2020 •

edited