Private package fails with `error: option --home not recognized` #240

bryantbiggs · 2018-09-13T21:41:34Z

I have a private repo on Github with a python package located under a subdirectory sdk/ that I can pip install locally with:

pip install -e git+ssh://git@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

Trying to add this to the requirments.txt, it fails with a permission denied - even with adding:

custom:
  pythonRequirements:
    dockerizePip: true
    dockerSsh: true

So I tried using a Git token to get around SSH issue for now; again it works locally with:

pip install -e git+ssh://<access-token>@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

However, now I am getting the error error: option --home not recognized

Output from SLS_DEBUG=* sls deploy --stage=dev:

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/requirements.txt in /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Obtaining sdk from git+https://<access-token>@github.com/<org>/<repo>.git@master#egg=sdk&subdirectory=sdk (from -r /var/task/requirements.txt (line 1))
  Cloning https://<access-token>@github.com/<org>/<repo>.git to ./src/sdk
Installing collected packages: sdk
  Running setup.py develop for sdk
    Complete output from command /var/lang/bin/python3.6 -c "import setuptools, tokenize;__file__='/var/task/src/sdk/sdk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-zkul1ujz:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --home not recognized

    ----------------------------------------
Command "/var/lang/bin/python3.6 -c "import setuptools, tokenize;__file__='/var/task/src/sdk/sdk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-zkul1ujz" failed with error code 1 in /var/task/src/sdk/sdk

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:247:11)
    at installRequirementsIfNeeded (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:423:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:502:29)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.11.4
     Serverless Version:     1.31.0

For reference, my setup.py:

#!/usr/bin/env python
"""
    Setup
    -----
    Enables the `setuptools` to build and install the library
"""

from setuptools import setup

packages = ['sdk']

requires = [
    'boto3~=1.9.2',
    'botocore~=1.12.2',
    'dnspython~=1.15.0',
    'email_validator~=1.0.3',
    'idna~=2.6',
    'requests~=2.19.1'
]
test_requirements = [
    'freezegun~=0.3.10',
    'pytest~=3.5.0',
    'pytest-cov'
]


with open('README.md', 'r', 'utf-8') as rfile:
    readme = rfile.read()

setup(
    name='sdk',
    version='0.1.0',
    description='Internal software development kit',
    long_description=readme,
    long_description_content_type='text/markdown',
    author='xxx team',
    author_email='admin@xxx.com',
    packages=packages,
    package_dir={'sdk': 'sdk'},
    python_requires=">=3.6",
    install_requires=requires,
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Natural Language :: English',
        'Topic :: Software Development :: Libraries',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3.6'
    ],
    tests_require=test_requirements
)

serverless: 1.31.1
serverless-python-requirements: 4.2.1

Any thoughts?

The text was updated successfully, but these errors were encountered:

AndrewFarley · 2018-09-14T08:59:34Z

What specifically did you add to your requirements.txt file? And if that repo is public, can you just share the actual repo url instead of hiding it so we can debug further?

dschep · 2018-09-14T13:28:17Z

What @AndrewFarley said. Do you have the -e in your requirements.txt? That almost definitely is an issue with the changes you made recently Andrew.

bryantbiggs · 2018-09-14T14:45:25Z

@dschep - yes, I have:

-e git+https://<access-token>@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

in my requirements.txt file (thats all I have in it for now while troubleshooting)

@AndrewFarley - unfortunately, it is not a public repo so I am unable to share the url. However, I have also tried replacing my private repo with the requests library in a very similar form under my requirements.txt file and it still gives an error. try using this url in place of my private url:

-e git+https://github.com/requests/requests.git@master#"egg=requests"

This works to install locally, but not within the serverless project with serverless-python-requirements

AndrewFarley · 2018-09-14T17:31:03Z

Hey guys, want a good laugh, this appears to be a pip/python bug. :P Check it out...

bash-4.2# python2.7 -m pip install --upgrade -t /var/task/ -r requirements.txt 
Obtaining measurements from git+https://github.com/jladan/package_demo.git#egg=measurements (from -r requirements.txt (line 3))
  Updating ./src/measurements clone
Collecting numpy (from measurements->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/c9/16/1134977cc35d2f72dbe80efa75a8e989ac21289f8e7e2c9005444cd17cd5/numpy-1.15.1-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, measurements
  Found existing installation: Measurements 0.5
    Uninstalling Measurements-0.5:
      Successfully uninstalled Measurements-0.5
  Running setup.py develop for measurements
    Complete output from command /usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/task/src/measurements/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-WMZs4g:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --home not recognized
    
    ----------------------------------------
  Rolling back uninstall of Measurements
Command "/usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/task/src/measurements/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-WMZs4g" failed with error code 1 in /var/task/src/measurements/

But if I remove the -t /var/task option...

bash-4.2# python2.7 -m pip install --upgrade -r requirements.txt 
Obtaining measurements from git+https://github.com/jladan/package_demo.git#egg=measurements (from -r requirements.txt (line 3))
  Updating ./src/measurements clone
Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib64/python2.7/site-packages (from measurements->-r requirements.txt (line 3)) (1.15.1)
Installing collected packages: measurements
  Found existing installation: Measurements 0.5
    Uninstalling Measurements-0.5:
      Successfully uninstalled Measurements-0.5
  Running setup.py develop for measurements
Successfully installed measurements

Thought it'd make you laugh a little like it did me. I did some extended testing, the -t /var/task/ is unnecessary... so I've removed it in an upcoming MR. I'm doing a little more testing (running all tests) to make sure it's fine and then I'll push it and MR it. So, fix incoming...

bryantbiggs · 2018-09-14T17:43:30Z

ah damn, good catch @AndrewFarley - looks like its been a well-documented bug in pip that they have been trying to reconcile pypa/pip#4390

I really appreciate the assistance and the package guys!

AndrewFarley · 2018-09-14T17:44:51Z

@bryantbiggs As mentioned here: pypa/pip#562 (comment) please try to remove the -e before your requirements. Seems to be a long standing pip/py bug. :P Confirmed it fixes it for me. Anyways, for something like this plugin, -e makes no sense because you wouldn't "edit" your files in a readonly environment (lambda). :P @dschep after author confirmed, I'd say safe to close this bug. Although, if you want, I/we can add a check to check for -e's with egg's and warn the user or something, eh? idk

dschep · 2018-09-14T18:12:42Z

Agreed, -e doesn't make much sense in a deployed package. A warning would be a good idea... or.... is there any reason the plugin couldn't just strip the -e?

AndrewFarley · 2018-09-14T18:25:47Z

Oh, it absolutely could! :) I tried removing /var/task and it doesn't work, I tried chdir-ing into the /var/task folder before running the command and it mostly worked but had some bugs. So, I think auto-stripping the -e makes a lot of sense. I'll MR that (instead) then, eh? The only problem might be that @dschep is that -e puts stuff in ./src/ and if people are expecting things there (which they shouldn't, but hey) then it will break stuff.

Sooo... maybe just a warning is best?

bryantbiggs · 2018-09-14T19:54:05Z

So I tried removing -e and no luck. To make it easier to communicate, I made a repro here https://github.com/bryantbiggs/repro-serverless-requirements that you can test out. Its very similar to the setup I am working with - I have my serverless stacks in one sub-directory and instead of symlinking a bunch of internal files to use in my stacks, i'd rather make an internal package and just pip install that so that it keeps my requirements the same, etc. Anyways, as you'll see in there - you can pip install the sdk locally, but the serverless-python-requirements doesn't seem to be able to pick it.

AndrewFarley · 2018-09-14T19:56:55Z

Alright thanks for that I’ll play with it a bit tomorrow and see if I can come up with a nice workaround.

By the way do you know if it worked on an older version of this plugin? Eg 4.1.0?

bryantbiggs · 2018-09-14T20:01:05Z

Thanks @AndrewFarley - I just tried v4.1.0 and v4.0.0 and neither of those worked. Different error message though:

Error --------------------------------------------------

  Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-8zynn1hw/sdk"


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-8zynn1hw/sdk"

    at installRequirements (/Users/B2/Downloads/repro-serverless-requirements/serverless/node_modules/serverless-python-requirements/lib/pip.js:114:11)
    at ServerlessPythonRequirements.installAllRequirements (/Users/B2/Downloads/repro-serverless-requirements/serverless/node_modules/serverless-python-requirements/lib/pip.js:202:5)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

AndrewFarley · 2018-09-14T20:16:23Z

We just have better debugging in the new version. Okay well at least the new stuff didn’t break this for you. That is good to hear. So I’m heading out but I’ll try to manage a fix for this tomorrow if I can

dschep · 2018-09-14T22:15:55Z

Yeah, now that you mention it @bryantbiggs I think this has never worked. See: #36

blbradley · 2019-01-04T23:01:40Z

Hey, I ran into this. Thanks for reporting.

nshgraph · 2019-02-05T17:55:29Z

Thanks, ran into this too.
Just a note, for some there is a reason to install in editable mode when developing and testing locally. This means we now have to have two different requirements files, one for deployment and one for development which seems unfortunate.
If it was possible to strip it out this seems a really good solution.

Previously, there were various with installing editable packages in different python / pip versions. serverless#240 This adds logic to strip out all '-e' editable flags from the requirements.txt file and issues a warning to the CLI. fixes serverless#240

dschep mentioned this issue Sep 14, 2018

Adding -f and -i to requirements.txt options ignore list before sorting #238

Merged

sjatkins mentioned this issue Sep 27, 2019

Question on private python modules ruzin/terraform_aws_lambda_python#1

Closed

jacksgt mentioned this issue Apr 19, 2020

Handle all requirement flags properly and strip -e flags #507

Merged

drice mentioned this issue May 8, 2020

Remove line associated with old poetry version #514

Merged

bsamuel-ui closed this as completed in #507 Oct 2, 2020

kichalla mentioned this issue Oct 6, 2020

Pip parameters cause "--home not recognized" error microsoft/Oryx#794

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Private package fails with `error: option --home not recognized` #240

Private package fails with `error: option --home not recognized` #240

bryantbiggs commented Sep 13, 2018 •

edited

AndrewFarley commented Sep 14, 2018

dschep commented Sep 14, 2018

bryantbiggs commented Sep 14, 2018 •

edited

AndrewFarley commented Sep 14, 2018

bryantbiggs commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018 •

edited

dschep commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018 •

edited

bryantbiggs commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018

bryantbiggs commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018

dschep commented Sep 14, 2018

blbradley commented Jan 4, 2019

nshgraph commented Feb 5, 2019

Private package fails with error: option --home not recognized #240

Private package fails with error: option --home not recognized #240

Comments

bryantbiggs commented Sep 13, 2018 • edited

AndrewFarley commented Sep 14, 2018

dschep commented Sep 14, 2018

bryantbiggs commented Sep 14, 2018 • edited

AndrewFarley commented Sep 14, 2018

bryantbiggs commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018 • edited

dschep commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018 • edited

bryantbiggs commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018

bryantbiggs commented Sep 14, 2018

AndrewFarley commented Sep 14, 2018

dschep commented Sep 14, 2018

blbradley commented Jan 4, 2019

nshgraph commented Feb 5, 2019

Private package fails with `error: option --home not recognized` #240

Private package fails with `error: option --home not recognized` #240

bryantbiggs commented Sep 13, 2018 •

edited

bryantbiggs commented Sep 14, 2018 •

edited

AndrewFarley commented Sep 14, 2018 •

edited

AndrewFarley commented Sep 14, 2018 •

edited