Generate requirements.txt from Pipfile.lock

[Zebradil]

How to generate requirements.txt file from existing Pipfile.lock without locking?
When I run pipenv lock -r it ignores existing Pipfile.lock and does locking process again.

There is a workaround for this:

$ pipenv sync
$ pipenv run pip freeze

In my particular situation I'm building docker image and using requirements.txt in Dockerfile. I'd like to avoid creating virtual environment on the host machine just to be able to create requirements.txt.

[frostming]

Pipenv doesn't provide a way for this, you can seek for other pipfile utility libraries such as pipfile-requirements

[anuj9196]

you can run

pipenv run pip freeze > requirements.txt

[Zebradil]

you can run

pipenv run pip freeze > requirements.txt

That's what I mentioned as a workaround in the first post.

But it works only if you have your pipenv environment synchronized (all packages are installed).
Extracting dependencies directly from Pipfile.lock is more convenient for me:

jq -r '.default
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements.txt

Blog post
jq tool

[frostming]

LOL, I have already mentioned that library.

[Zebradil]

I, personally, dislike to have a dedicated library for that. Also, there is a higher chance that a team member already has jq or some other general purpose tool installed.

[anuj9196]

you can even run

 pipenv lock --requirements > requirements.txt

[Zebradil]

It will not work as you expect, because, as I wrote:

When I run pipenv lock -r it ignores existing Pipfile.lock and does locking process again.

In other words, it performs an update, which potentially can destroy a distribution. Imagine, you generate requirements.txt to use it in Dockerfile to build a docker image. Locally your application works, but when you generate requirements.txt using pipenv lock, requirements might get updated to incompatible or just broken versions (hopefully, it's a rare case, though). And you will not know this before running the image. So, you'll need to test the app again after running pipenv lock.

If you don't want to use jq, then better use the approach I proposed in the first post with pipenv sync (which doesn't do update).

[ye]

@Zebradil your jq oneliner approach is much simpler than @frostming 's own pipfile-requirements package (100+ lines python code) since I've already installed jq, no other dependencies required, which is great.

However, after a few git commits I noticed the difference between what pipenv lock --requirements outputs and what jq gleans through Pipfile.lock file and prints out:

• jq output doesn't have -i https://pypi.org/simple as the very first line, as opposed to what pipenv lock --r always insert as the very first line.
• jq output doesn't include the annotation for packages. For example: pipenv lock --r output has this lineappnope==0.1.0 ; sys_platform == 'darwin', but in jq output, it's appnope==0.1.0. Another example is pipenv lock -r generates pexpect==4.7.0 ; sys_platform != 'win32' whereas jq generates pexpect==4.7.0, not sure if this matters or not.
• the requirements file package ordering is different between the two outputs. jq which presumably takes the package order in Pipfile.lock file, which always sorts by ascending order in alphabets and character length, for example, flask is in front of flask-sqlalchemy or any flask-XXXXX packages, whereas pipenv lock --r outputs flask behind flask-sqlalchemy, which is different from the order in the Pipfile.lock. This is a major annoyance because it doesn't generate a minimum git diff. I would consider this is a bug in pipenv.

[Zebradil]

Hi @ye, nice comparison of the methods. It might help people choosing proper solution for their particular situation and avoid caveats.

Yes, as you said, the proposed approach with jq has limited functionality. It is possible to extend it to add annotations and package index URL, but I have no need for this right now.

To avoid having differences in generated requirements.txt one should consider using the same approach every time. In a similar way, using different code formatting tools might lead to inconsistent results. So, I don't see a problem here.

[rcastill]

you can run

pipenv run pip freeze > requirements.txt

That's what I mentioned as a workaround in the first post.

But it works only if you have your pipenv environment synchronized (all packages are installed).
Extracting dependencies directly from Pipfile.lock is more convenient for me:

jq -r '.default
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements.txt

Blog post
jq tool

Hi,

Thanks for your solution. I ran into the same problem but I also need the definition of sources created by pipenv lock -r, i.e.: -i, --extra-index-url. This is because I work with private sources.

@Zebradil I think you mentioned that.

So I created yet another minimal no-dependencies script in python that includes that functionality. It also expands env vars in case that you have your sources defined that way in your Pipfile.

If anybody wants to take a look I'll leave it here: https://gist.github.com/rcastill/dab85c234dd10fa7af56755116c75aee

[dms-cat]

In case it helps anyone else, here's how to include the hashes in the results:

 jq --raw-output '.default | to_entries[] | .key + .value.version + (.value.hashes | map(" --hash=\(.)") | join(""))' Pipfile.lock

This creates entries like

paramiko==2.6.0 --hash=sha256:99f0179bdc176281d21961a003ffdb2ec369daac1a1007241f53374e376576cf --hash=sha256:f4b2edfa0d226b70bd4ca31ea7e389325990283da23465d572ed1f70a7583041

Which makes pip enforce the hashes.

If you want to include only the requirements which were in the original requirements file (provided they were already locked to a specific version with ==):

 jq --raw-output '.default | to_entries[] | .key + .value.version + (.value.hashes | map(" --hash=\(.)") | join(""))' Pipfile.lock | grep --file=<(grep --only-matching --perl-regexp '^.*(?===)' requirements.txt | tr '[:upper:]' '[:lower:]') > new.txt && mv new.txt requirements.txt

tr is necessary because requirements.txt files may contain mixed case package names but pipenv install -r requirements.txt lowercases them in Pipfile.

[BeyondEvil]

Here's a small python script in case you want to turn the Pipfile (not the lock-file) into a requirements.txt file.

import configparser


def main():
    parser = configparser.ConfigParser()
    parser.read("Pipfile")

    packages = "packages"
    with open("requirements.txt", "w") as f:
        for key in parser[packages]:
            value = parser[packages][key]
            f.write(key + value.replace("\"", "") + "\n")


if __name__ == "__main__":
    main()

[linusguan]

@frostming Hi, I found https://github.com/frostming/pipfile-requirements useful but why it didn't get integrated into pipenv?

[frostming]

@linusguan The tool exists for those who don't want to install the big pipenv library, when you have installed pipenv, you can use pipenv lock -r

[linusguan]

@frostming I find it quite useful for use with other tools that do not support pipfile.lock.
The problem with pipenv lock -r is it updates pipfile.lock so I cannot use it to produce deterministic build along with other tools. Something like pipenv lock -r --ignore-pipfile would be ideal.

[sergeyt]

Here is yet another python script to generate requirements.txt from Pipfile.lock file with hashes:

import os
import json

__dir__ = os.path.dirname(os.path.realpath(__file__))


def read_json_file(path):
    with open(path) as f:
        return json.load(f)


def main():
    root = read_json_file(os.path.join(__dir__, 'Pipfile.lock'))

    for name, pkg in root["default"].items():
        version = pkg["version"]
        sep = lambda i: "" if i == len(pkg["hashes"]) - 1 else " \\"
        hashes = [f'--hash={t}{sep(i)}' for i, t in enumerate(pkg["hashes"])]
        tail = '' if len(hashes) == 0 else f' {hashes[0]}'
        print(f'{name} {version}{tail}')
        for h in hashes[1:]:
            print(f'    {h}')


if __name__ == "__main__":
    main()

[harshit98]

@Zebradil Thanks! Your solution really worked for me.

• I first install jq tool using brew install jq
• Then used your script to generate requirements.txt from Pipfile.lock

[jacobeatsspam]

It looks like this can be solved with the --keep-outdated flag, or am I mistaken?

pipenv lock --keep-outdated -d -r > requirements.txt

P.S. that's an annoyingly verbose flag to solve this

[paytonrules]

Unfortunately @jacobisaliveandwell the --keep-outdated flag appears to update subdependencies: #3975

[jacobeatsspam]

@paytonrules That's a bug, but the spirit of the flag is still the answer to this issue.

P.S. No need to thumbs down for that :-(

[Darkless012]

Just want to mention that from stated workarounds, there are differences:
pipenv run pip freeze returns case-sensitive package names (eg. PyYAML)
pipenv lock --requirements returns all lower-case package names (eg. pyyaml)

[earthgecko]

Pure bash, just packages, nothing else, in case someone cannot or does not want to install jq, just in case it helps someone,

cat Pipfile.lock \
  | grep -B1 '"hashes"\|"version": \|"extras":' \
  | grep -v '"markers": \|"hashes": ' \
  | grep ": {\|version" \
  | sed -e 's/: {$//g' \
  | tr '\n' ',' | tr -s ' ' ' ' \
  | sed -e 's/, "version": "//g;s/", "/ /g;s/"//g;s/,//g' \
  | tr ' ' '\n' \
| grep -v "^$" > requirements.txt

Updated to handles extras as per @vincentlepage comment below #3493 (comment)

[westurner]

Is there anything that also copies the hashes from the Pipfile to requirements.txt (e.g. given something like a platform str) so that pip install --require-hashes works?

• https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
• https://pip.pypa.io/en/stable/reference/pip_hash/

$ pip install --help
# ...
  --require-hashes            Require a hash to check each requirement against, for repeatable installs. This option is implied when any package in a
                              requirements file has a --hash option.

[fridex]

You can use micropipenv that can convert Pipenv.lock (also poetry.lock) files to requirements.txt (raw requirements.txt or pip-tools compatible). See https://github.com/thoth-station/micropipenv/#micropipenv-requirements--micropipenv-req

[vanschelven]

As mentioned in the blog post mentioned above, but not in this thread yet, for development dependencies you need another one-liner if you're going the jq way.

jq -r '.develop
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements-dev.txt

[vincentlepage]

Pure bash, just packages, nothing else, in case someone cannot or does not want to install jq, just in case it helps someone,

cat Pipfile.lock \
  | grep -B1 '"hashes"\|"version": ' \
  | grep -v '"markers": \|"hashes": ' \
  | grep ": {\|version" \
  | sed -e 's/: {$//g' \
  | tr '\n' ',' | tr -s ' ' ' ' \
  | sed -e 's/, "version": "//g;s/", "/ /g;s/"//g;s/,//g' \
  | tr ' ' '\n' \
| grep -v "^$" > requirements.txt

Works almost perfectly, though I had an issue with some "extras".
A minor addition to correct the issue:

cat Pipfile.lock
| grep -B1 '"hashes"|"version"|"extras": '
| grep -v '"markers": |"hashes": '
| grep ": {|version"
| sed -e 's/: {$//g'
| tr '\n' ',' | tr -s ' ' ' '
| sed -e 's/, "version": "//g;s/", "/ /g;s/"//g;s/,//g'
| tr ' ' '\n'
| grep -v "^$" > requirements.txt

[ZelphirKaltstahl]

I just saw this issue in my e-mail and it made me wonder: Seriously? Couldn't you use a part of pipenv via Python to read in the lock file and get some kind of data structure, which contains the version numbers you are looking for and then write them to a simple requirements file?

That seems like a much more robust approach to me, than the script posted. However, I am not even sure Pipenv allows for such a way, so it might just be a fantasy on my part.

[Zebradil]

I'm using poetry instead of pipenv whenever I can — that solves the problem. Not sure if there are any reasons to use pipenv nowadays, except maybe in older projects, where the team is against that.

[ZelphirKaltstahl]

I'm using poetry instead of pipenv whenever I can — that solves the problem. Not sure if there are any reasons to use pipenv nowadays, except maybe in older projects, where the team is against that.

To be fair, both tools have their share of issues. For a long time Poetry seemed to only use a single core for all installation of packages into an environment, while Pipenv already used multiple cores. Then there are problems installing some packages on either of the two tools. Something always comes up when working with either one of them. I see regressions in issues of both tools. Simple things like parsing version numbers of packages fails. I think both tools are already valuable, as I have used both in real projects, but still have a some way to go in terms of code quality and testing. I appreciate, that people are working on both tools and I guess only time will tell, which one will be more used.

[glensc]

pipenv lock --requirements / pipenv lock -r is no longer there. any ideas why?

➔ pipenv lock -r
Loading .env environment variables...
Usage: pipenv lock [OPTIONS]
Try 'pipenv lock -h' for help.

Error: No such option: -r

➔ pipenv --version
pipenv, version 2022.9.21

[matteius]

@glensc it was replaced by the standalone requirements command.

[glensc]

ok, found this:

• Implements requirements command as per #4959 #5013

  Generate a requirements.txt file (including dev):
       $ pipenv requirements --dev > requirements.txt

[Acel-01]

you can run

pipenv run pip freeze > requirements.txt

Thanks

[matteius]

Actually pipenv requirements command generates a requirements.txt from the Pipfile.lock -- pipenv run pip freeze generates it based on what you have installed only.