`pip install --upgrade azure-cli` fails on Cygwin due to psutil #9399

ElvenSpellmaker · 2019-05-15T07:15:25Z

Describe the bug
I tried to upgrade azure-cli from 2.0.31 to the latest (2.0.62 as of writing)

To Reproduce
pip install --upgrade azure-cli

Expected behavior
az should upgrade.

Environment summary
Windows 10
Cygwin (latest)

Additional context

Collecting psutil>=5.6.1 (from azure-cli-core==2.0.64->azure-cli)
  Using cached https://files.pythonhosted.org/packages/c6/c1/beed5e4eaa1345901b595048fab1c85aee647ea0fc02d9e8bf9aceb81078/psutil-5.6.2.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: platform cygwin is not supported
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-NwRvbi/psutil/

The text was updated successfully, but these errors were encountered:

yugangw-msft · 2019-05-15T15:44:20Z

@adewaleo, the psutil was pulled in for the new az feedback, anyway we can find an alternative?

ElvenSpellmaker · 2019-05-24T10:20:10Z

~~2.0.65 seems to install under Cygwin now, but has problems running everything:~~

~~Should I open a new issue for this? Thanks.~~

ElvenSpellmaker · 2019-05-24T10:44:58Z

Weirdly enough I uninstalled all traces of the azure-cli (pip list | awk '{print $1}' | grep '^azure' | xargs pip uninstall) and tried re-installing and 2.0.65 complains about psutil again... 🤔

ElvenSpellmaker · 2019-05-24T10:53:53Z

Also it seems I can't even install an older version any more as the ps-util breaks that.

Collecting psutil>=5.6.1 (from azure-cli-core->azure-cli==2.0.31)
  Using cached https://files.pythonhosted.org/packages/c6/c1/beed5e4eaa1345901b595048fab1c85aee647ea0fc02d9e8bf9aceb81078/psutil-5.6.2.tar.gz

adewaleo · 2019-05-24T17:09:14Z

@ElvenSpellmaker, Yes, unfortunately it turns out that psutil does not support cygwin yet. This is sad as it makes it very easy for az feedback to detect the shell the cli command was executed in.

This sprint, I will remove it as a dependency and try to find other ways to detect what shell a user is running. I am not sure how to detect for certain that a command was executed from powershell aside from checking that the grandparent process was indeed powershell.exe

For now there are two workarounds that come to mind:

Install version 2.0.60 of the CLI. I believe az feedback was updated around 2.0.61 or so
Install the azure-cli without the psutil dependencies. An approach could be the following:
- in a non cygwin environment run pip install azure-cli.
- run pip freeze > requirements.txt
- remove the psutil dependency from the file
- pip install -r requirements.txt

ElvenSpellmaker · 2019-05-24T17:36:05Z

Unfortunately as I've stated above, it's a dependency that tries to install the psutil and so I can't install an older version even! =(

adewaleo · 2019-05-24T18:35:02Z

Unfortunately as I've stated above, it's a dependency that tries to install the psutil and so I can't install an older version even! =(

installing 2.0.50 should solve the problem as a work around. az feedback only started using psutil recently. We didn't always have this dependency.

I was wrong to ask you to install 2.0.62, the change was made 4 or so sprints ago.

ElvenSpellmaker · 2019-05-25T20:41:30Z

Unfortunately installing older versions has the same effect, it always tries to pull in psutil

Requirement already satisfied: futures; python_version < "3.0" in /usr/lib/python2.7/site-packages (from azure-cli-core->azure-cli==2.0.50) (3.2.0)
Collecting psutil>=5.6.1 (from azure-cli-core->azure-cli==2.0.50)
  Using cached https://files.pythonhosted.org/packages/c6/c1/beed5e4eaa1345901b595048fab1c85aee647ea0fc02d9e8bf9aceb81078/psutil-5.6.2.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: platform cygwin is not supported
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-mhvWZc/psutil/

dtzar · 2019-05-28T17:44:37Z

Not the same error, but related to psutil with a Docker build regardless of using 2.0.46 or 2.0.65
https://dev.azure.com/csedevops/MLOps/_build/results?buildId=757 (2.0.46)
https://dev.azure.com/csedevops/MLOps/_build/results?buildId=760 (2.0.65)

This is there because psutil requires gcc to be present to compile the psutil binary. After I install gcc it completes ok. IMO there should be no requirement for gcc to pip install azure-cli

  building 'psutil._psutil_linux' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/psutil
  gcc -pthread -B /usr/local/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=562 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.7m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.7/psutil/_psutil_common.o
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for psutil‌
  Running setup.py clean for psutil‌
Successfully built vsts-cd-manager sshtunnel pydocumentdb pyyaml pyperclip tabulate antlr4-python3-runtime
Failed to build psutil

adewaleo · 2019-05-28T23:39:08Z

Not the same error, but related to psutil with a Docker build regardless of using 2.0.46 or 2.0.65
https://dev.azure.com/csedevops/MLOps/_build/results?buildId=757 (2.0.46)
https://dev.azure.com/csedevops/MLOps/_build/results?buildId=760 (2.0.65)

  building 'psutil._psutil_linux' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/psutil
  gcc -pthread -B /usr/local/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=562 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.7m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.7/psutil/_psutil_common.o
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for psutil‌
  Running setup.py clean for psutil‌
Successfully built vsts-cd-manager sshtunnel pydocumentdb pyyaml pyperclip tabulate antlr4-python3-runtime
Failed to build psutil

What docker file are you using to build the docker image? Are you using the one in our repo?

There was an added dependency when psutil was aded to our repo.

@dtzar

dtzar · 2019-05-29T06:36:19Z

https://hub.docker.com/r/dtzar/conda3mlops

adewaleo · 2019-05-29T21:18:34Z

https://hub.docker.com/r/dtzar/conda3mlops

In our dockerfile, I had to add the linux-headers dependency for psutil to work. There might be a similar case where you need to include (a dependency) with gcc in your docker file

see the following for more info: giampaolo/psutil#1143

adewaleo · 2019-07-17T05:56:40Z

Hey @ElvenSpellmaker can you try running pip install -r requirements.py3.windows.txt, but remove psutil's dependency from the file beforehand.

This should be a work around for cygwin. Psutil is only used in feedback. I could modify az feedback to not crash if psutil is not installed.

Let me know if it works

adewaleo · 2019-07-26T21:22:14Z

I will be closing this issue as there is a workaround

ElvenSpellmaker · 2019-08-02T12:46:13Z

@adewaleo Hey, sorry for my late response, I get this error when using the requirements for Python 2:

I'm wondering if I can just remove that too

lweberru · 2019-11-22T12:04:13Z

I could install it via virtualenv. But where is the installation than. How can I start it after :
azdev setup -c .

ElvenSpellmaker · 2019-11-22T12:05:15Z

If you're in the virtual env you should be able to type az.

Each terminal window will need that virtualenv source'd in to find the correct binary

lweberru · 2019-11-22T12:12:14Z

I have to start it directly within the installed directory with a './az'. Else it does not work. I tried to link the file to /usr/bin/az, but then it fails
An also it fails for the command az webapp list:
The command failed with an unexpected error. Here is the traceback:

No module named 'fabric'
Traceback (most recent call last):
File "/home/lukas/AzureCliInstall/azure-cli/lib/python3.7/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/commands/init.py", line 523, in execute
self.commands_loader.load_arguments(command)
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/init.py", line 293, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/commands/init.py", line 291, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/home/lukas/AzureCliInstall/azure-cli/lib/python3.7/site-packages/knack/commands.py", line 97, in load_arguments
cmd_args = self.arguments_loader()
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/init.py", line 490, in default_arguments_loader
op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/init.py", line 530, in get_op_handler
op = import_module(mod_to_import)
File "/home/lukas/AzureCliInstall/azure-cli/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli/azure/cli/command_modules/appservice/custom.py", line 25, in
from fabric import Connection
ModuleNotFoundError: No module named 'fabric'

ElvenSpellmaker · 2019-11-22T12:13:56Z

If you activated the virtualenv correctly then you'l have (env) at the beginning of your prompt and the bin directory for that virtalenv will be in your path so just typing az should work.

lweberru · 2019-11-22T12:15:43Z

Probably i failed with this virtualenv as I even dont know what it is

ElvenSpellmaker · 2019-11-22T12:18:15Z

Virtualenv is a virtual environment (see https://docs.python.org/3/library/venv.html), rather than installing modules globally you install them to a separate directory, kind of like how PHP's composer, Ruby's bundler, and Node.js's npm.

EDIT:

composer installs to vendor in each project
bundler installs globally by default (but can go to a vendor folder), and works out deps at runtime by using the Gemfile/Gemfile.lock
npm installs to node_modules for each project

lweberru · 2019-11-22T12:19:16Z

Which command I have to execute to do this virtualenv thing correctly?

ElvenSpellmaker · 2019-11-22T12:21:12Z

python3 -m venv env
source env/bin/activate

lweberru · 2019-11-22T12:22:56Z

$ python3 -m venv env
(azure-cli)
lukas@LUKAS ~/AzureCliInstall/azure-cli
$ source env/bin/activate
(env)
lukas@LUKAS ~/AzureCliInstall/azure-cli
$ az
-bash: az: Kommando nicht gefunden.
(env)
lukas@LUKAS ~/AzureCliInstall/azure-cli

$ echo $PATH
/home/lukas/AzureCliInstall/azure-cli/env/bin:/usr/local/bin:/usr/bin:...

and the az binary resides in
/home/lukas/AzureCliInstall/azure-cli/src/azure-cli/az

ElvenSpellmaker · 2019-11-22T12:25:34Z

Now try azdev setup -c . inside the repo directory root and then try az again

lweberru · 2019-11-22T13:03:19Z

$ azdev setup -c .

=======================
| Azure CLI Dev Setup |

Azure CLI:
/home/lukas/AzureCliInstall/azure-cli

Installing packages

Upgrading pip...
Installing requirements.txt...
Installing azure-cli-nspkg...
Installing azure-cli-telemetry...
Installing azure-cli-core...
Installing azure-cli...
Installing azure-cli-testsdk...
Installing azure-nspkg...
Installing azure-mgmt-nspkg...

Elapsed time: 18 min 31 sec

Finished dev setup!

(env)
lukas@LUKAS ~/AzureCliInstall/azure-cli
$ az
-bash: az: Kommando nicht gefunden.
(env)

ElvenSpellmaker · 2019-11-22T13:43:57Z

and the az binary resides in
/home/lukas/AzureCliInstall/azure-cli/src/azure-cli/az

It should reside in env/bin

lweberru · 2019-11-22T14:34:08Z

Yes and that seems to be the problem...

adewaleo · 2019-11-22T22:31:37Z

@ElvenSpellmaker thanks a lot for helping out @lweberru

lweberru · 2019-11-23T10:22:13Z

So whatever I do, the binaries are always inside
/home/lukas/AzureCliInstall/azure-cli/src/azure-cli/az
and the path (virtual env) is
/home/lukas/AzureCliInstall/azure-cli/env/bin

I also tried to mv or copy the files out of /home/lukas/AzureCliInstall/azure-cli/src/azure-cli/ to the virtual env but that also fails. Any idea?

lweberru · 2019-11-23T10:23:20Z

my commands are
1.) python3 -m venv env
2.) source env/bin/activate
3.) azdev setup -c .

then i test with a single "az"

lweberru · 2019-11-23T11:02:45Z

But even more relevant is, that the command az itself does not work:

The command failed with an unexpected error. Here is the traceback:

No module named 'fabric'
Traceback (most recent call last):
File "/home/lukas/AzureCliInstall/azure-cli/env/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/commands/init.py", line 523, in execute
self.commands_loader.load_arguments(command)
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/init.py", line 293, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/commands/init.py", line 291, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/home/lukas/AzureCliInstall/azure-cli/env/lib/python3.6/site-packages/knack/commands.py", line 97, in load_arguments
cmd_args = self.arguments_loader()
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/init.py", line 490, in default_arguments_loader
op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli-core/azure/cli/core/init.py", line 530, in get_op_handler
op = import_module(mod_to_import)
File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/lukas/AzureCliInstall/azure-cli/src/azure-cli/azure/cli/command_modules/appservice/custom.py", line 25, in
from fabric import Connection
ModuleNotFoundError: No module named 'fabric'

only "az login" works

adewaleo · 2019-11-25T20:08:31Z

does pip list / freeze include fabric==2.4.0 or similar?

lweberru · 2019-11-26T17:01:57Z

No it does not. Can I install it seperatly?

lweberru · 2019-11-26T17:04:37Z

I installed fabric via "pip install fabric"
pip install fabric
[...]
Successfully installed fabric-2.5.0 invoke-1.3.0

and now I get:
az webapp list
[...]
ModuleNotFoundError: No module named 'azure.mgmt.storage'

I installed that module as well. This is than followed by the next and next and next module, which is missing...

adewaleo · 2019-11-26T18:42:44Z

You can but the point of running azdev setup -c . is to install these dependencies from the project's setup.py... its best to get that working or you might have to manually install the missing dependencies

ElvenSpellmaker · 2019-11-26T18:43:48Z

I think the first thing to work out is why it's building to ./ rather than ./env/bin... 🤔

adewaleo · 2019-11-26T18:58:28Z

after azdev setup, within a virtual env, you should be able to run az automatically.

What does your path show in the virtualenv. It should be prefixed by the path to the binaries in your virtualenv after activating it.

On my mac It shows:

$ echo $PYTHONPATH
http://users/tosin/Repos/my_venvs/cli-venv/bin:.......:/usr/local/go/bin

avoidik · 2020-03-27T20:45:55Z

after #11665 has been merged it is possible to install latest azure-cli in cygwin

ElvenSpellmaker · 2020-03-27T20:57:15Z

That's great news, let's hope it makes a release soon! @avoidik Could you update this issue when it makes a release please?

avoidik · 2020-03-27T21:25:30Z

@ElvenSpellmaker it is already in there

$ az version | jq -r 'with_entries(select(.key=="azure-cli"))'
This command is in preview. It may be changed/removed in a future release.
{
  "azure-cli": "2.2.0"
}

ElvenSpellmaker · 2020-03-27T21:31:18Z

Thanks!

yugangw-msft assigned adewaleo May 15, 2019

yugangw-msft added the Feedback az feedback label May 15, 2019

adewaleo added this to the Sprint 62 milestone May 15, 2019

adewaleo modified the milestones: Sprint 62, Sprint 63 May 31, 2019

adewaleo modified the milestones: Sprint 63, Sprint 65 Jun 14, 2019

adewaleo modified the milestones: Sprint 65, Sprint 66 Jul 11, 2019

tjprescott added Installation Feedback az feedback and removed Feedback az feedback Installation labels Jul 17, 2019

adewaleo closed this as completed Jul 26, 2019

adewaleo added the Workaround label Jul 26, 2019

avoidik mentioned this issue Dec 25, 2019

{Packaging} Get rid of psutil dependency #11665

Merged

pspag mentioned this issue Oct 4, 2020

{Core} Add SessionId for telemetry #15076

Merged

jiasli mentioned this issue Mar 16, 2022

{Packaging} Use proper PEP 508 environment marker for dependencies #21660

Merged

pip install --upgrade azure-cli fails on Cygwin due to psutil #9399

pip install --upgrade azure-cli fails on Cygwin due to psutil #9399

Comments

ElvenSpellmaker commented May 15, 2019

yugangw-msft commented May 15, 2019

ElvenSpellmaker commented May 24, 2019 • edited

ElvenSpellmaker commented May 24, 2019

ElvenSpellmaker commented May 24, 2019

adewaleo commented May 24, 2019 • edited

ElvenSpellmaker commented May 24, 2019

adewaleo commented May 24, 2019 • edited

ElvenSpellmaker commented May 25, 2019

dtzar commented May 28, 2019 • edited

adewaleo commented May 28, 2019 • edited

dtzar commented May 29, 2019

adewaleo commented May 29, 2019 • edited

adewaleo commented Jul 17, 2019

adewaleo commented Jul 26, 2019

ElvenSpellmaker commented Aug 2, 2019 • edited

lweberru commented Nov 22, 2019

ElvenSpellmaker commented Nov 22, 2019

lweberru commented Nov 22, 2019

ElvenSpellmaker commented Nov 22, 2019

lweberru commented Nov 22, 2019 • edited

ElvenSpellmaker commented Nov 22, 2019 • edited

lweberru commented Nov 22, 2019

ElvenSpellmaker commented Nov 22, 2019

lweberru commented Nov 22, 2019 • edited

ElvenSpellmaker commented Nov 22, 2019

lweberru commented Nov 22, 2019

======================= | Azure CLI Dev Setup |

Installing packages

Finished dev setup!

ElvenSpellmaker commented Nov 22, 2019 • edited

lweberru commented Nov 22, 2019

adewaleo commented Nov 22, 2019

lweberru commented Nov 23, 2019

lweberru commented Nov 23, 2019

lweberru commented Nov 23, 2019

adewaleo commented Nov 25, 2019

lweberru commented Nov 26, 2019

lweberru commented Nov 26, 2019 • edited

adewaleo commented Nov 26, 2019

ElvenSpellmaker commented Nov 26, 2019

adewaleo commented Nov 26, 2019 • edited

avoidik commented Mar 27, 2020

ElvenSpellmaker commented Mar 27, 2020

avoidik commented Mar 27, 2020

ElvenSpellmaker commented Mar 27, 2020

`pip install --upgrade azure-cli` fails on Cygwin due to psutil #9399

`pip install --upgrade azure-cli` fails on Cygwin due to psutil #9399

ElvenSpellmaker commented May 24, 2019 •

edited

adewaleo commented May 24, 2019 •

edited

adewaleo commented May 24, 2019 •

edited

dtzar commented May 28, 2019 •

edited

adewaleo commented May 28, 2019 •

edited

adewaleo commented May 29, 2019 •

edited

ElvenSpellmaker commented Aug 2, 2019 •

edited

lweberru commented Nov 22, 2019 •

edited

ElvenSpellmaker commented Nov 22, 2019 •

edited

lweberru commented Nov 22, 2019 •

edited

=======================
| Azure CLI Dev Setup |

ElvenSpellmaker commented Nov 22, 2019 •

edited

lweberru commented Nov 26, 2019 •

edited

adewaleo commented Nov 26, 2019 •

edited