Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTPError: 400 Only one sdist may be uploaded per release (when using --skip-existing) #15824

Open
nanonano opened this issue Apr 19, 2024 · 4 comments
Open

HTTPError: 400 Only one sdist may be uploaded per release (when using --skip-existing) #15824

nanonano opened this issue Apr 19, 2024 · 4 comments
Labels
blocked Issues we can't or shouldn't get to yet bug 🐛

Comments

@nanonano
Copy link

Describe the bug
Since 15th April my uploads to test.pypi.org have started failing with the error:

HTTPError: 400 Only one sdist may be uploaded per release.

Not sure if something has changed, I thought if the --skip-existing flag was set this was okay?

Below is the GitHub actions log output.

Run twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[2](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:2)
  twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[3](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:3)
  shell: /usr/bin/bash -e {0}
[4](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:4)
  env:
[5](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:5)
    pythonLocation: /opt/hostedtoolcache/Python/3.12.3/x[6](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:6)4
6
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.3/x64/lib/pkgconfig
[7](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:7)
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
[8](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:8)
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
[9](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:9)
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
[10](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:10)
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.3/x64/lib
[11](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:11)
    TWINE_USERNAME: __token__
[12](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:12)
    TWINE_PASSWORD: ***
[13](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:14)
Uploading distributions to https://test.pypi.org/legacy/
[14](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:15)
Uploading BittyTax-0.5.3.dev3-py3-none-any.whl
[15](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:16)
25l
[16](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:17)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/935.3 kB • --:-- • ?
[17](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:18)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/935.3 kB • --:-- • ?
[18](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:19)
 28% ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 262.1/935.3 kB • 00:01 • 3.1 MB/s
[19](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:20)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[20](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:21)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[21](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:22)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[22](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:23)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[23](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:24)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[24](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:25)
25hWARNING  Skipping BittyTax-0.5.3.dev3-py3-none-any.whl because it appears to    
[25](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:26)
         already exist                                                          
[26](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:27)
Uploading bittytax-0.5.3.dev3.tar.gz
[27](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:28)
25l
[28](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:29)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/910.7 kB • --:-- • ?
[29](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:30)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 910.7/910.7 kB • 00:00 • 224.2 MB/s
[30](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:31)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 910.7/910.7 kB • 00:00 • 224.2 MB/s
[31](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:32)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 910.7/910.7 kB • 00:00 • 224.2 MB/s
[32](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:33)
25hWARNING  Error during upload. Retry with the --verbose option for more details. 
[33](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:34)
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
[34](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:35)
         Only one sdist may be uploaded per release.                            
[35](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:36)
Error: Process completed with exit code 1.

Expected behavior

I would expect just the warning as before:

WARNING Skipping BittyTax-0.5.3.dev3.tar.gz because it appears to already exist

Run twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[2](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:2)
  twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[3](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:3)
  shell: /usr/bin/bash -e {0}
[4](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:4)
  env:
[5](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:5)
    pythonLocation: /opt/hostedtoolcache/Python/3.12.2/x[6](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:6)4
6
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.2/x64/lib/pkgconfig
[7](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:7)
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
[8](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:8)
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
[9](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:9)
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
[10](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:10)
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.2/x64/lib
[11](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:11)
    TWINE_USERNAME: __token__
[12](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:12)
    TWINE_PASSWORD: ***
[13](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:14)
Uploading distributions to https://test.pypi.org/legacy/
[14](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:15)
Uploading BittyTax-0.5.3.dev3-py3-none-any.whl
[15](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:16)
25l
[16](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:17)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/928.6 kB • --:-- • ?
[17](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:18)
 65% ━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 606.2/928.6 kB • 00:01 • 30.0 MB/s
[18](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:19)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[19](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:20)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[20](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:21)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[21](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:22)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[22](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:23)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[23](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:24)
25hWARNING  Skipping BittyTax-0.5.3.dev3-py3-none-any.whl because it appears to    
[24](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:25)
         already exist                                                          
[25](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:26)
Uploading BittyTax-0.5.3.dev3.tar.gz
[26](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:27)
25l
[27](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:28)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/905.3 kB • --:-- • ?
[28](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:29)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 905.3/905.3 kB • 00:00 • 155.5 MB/s
[29](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:30)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 905.3/905.3 kB • 00:00 • 155.5 MB/s
[30](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:31)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 905.3/905.3 kB • 00:00 • 155.5 MB/s
[31](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:32)
25hWARNING  Skipping BittyTax-0.5.3.dev3.tar.gz because it appears to already exist

To Reproduce

My Platform

Current runner version: '2.315.0'
[2](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:2)
Operating System
[3](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:3)
  Ubuntu
[4](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:4)
  22.04.4
[5](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:5)
  LTS
[6](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:7)
Runner Image
[7](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:8)
  Image: ubuntu-22.04
[8](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:9)
  Version: 20240407.1.0
[9](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:10)
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240407.1/images/ubuntu/Ubuntu2204-Readme.md
[10](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:11)
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240407.1

Additional context
None
@nanonano nanonano added bug 🐛 requires triaging maintainers need to do initial inspection of issue labels Apr 19, 2024
@di
Copy link
Member

di commented Apr 19, 2024 •
edited

The issue here seems to be the difference between bittytax-0.5.3.dev3.tar.gz (which is what you're trying to upload) and BittyTax-0.5.3.dev3.tar.gz (what you've already uploaded).

Packaging considers these the same file, because:

>>> from packaging.utils import parse_sdist_filename
>>> parse_sdist_filename('bittytax-0.5.3.dev3.tar.gz') == parse_sdist_filename('BittyTax-0.5.3.dev3.tar.gz')
True

but it looks like the logic in twine doesn't take this normalization into account: https://github.com/pypa/twine/blob/67e87ef4221123454403b11ee8f802e87fcc13fd/twine/repository.py#L229

...so it doesn't correctly skip the upload and attempts to do it anyways.

Our error message here is a little confusing though, because we also don't take normalization into account when checking for duplicate filenames:

warehouse/warehouse/forklift/legacy.py

Lines 910 to 939 in 5a2d6f8

# Check to see if the file that was uploaded exists already or not.
is_duplicate = _is_duplicate_file(request.db, filename, file_hashes)
if is_duplicate:
request.tm.doom()
return HTTPOk()
elif is_duplicate is not None:
raise _exc_with_message(
HTTPBadRequest,
# Note: Changing this error message to something that doesn't
# start with "File already exists" will break the
# --skip-existing functionality in twine
# ref: https://github.com/pypi/warehouse/issues/3482
# ref: https://github.com/pypa/twine/issues/332
"File already exists. See "
+ request.help_url(_anchor="file-name-reuse")
+ " for more information.",
)
# Check to see if the file that was uploaded exists in our filename log
if request.db.query(
request.db.query(Filename).filter(Filename.filename == filename).exists()
).scalar():
raise _exc_with_message(
HTTPBadRequest,
"This filename has already been used, use a "
"different version. "
"See "
+ request.help_url(_anchor="file-name-reuse")
+ " for more information.",
)

And so this fails on our check for multiple source distributions instead:

warehouse/warehouse/forklift/legacy.py

Lines 941 to 953 in 5a2d6f8

# Check to see if uploading this file would create a duplicate sdist
# for the current release.
if (
form.filetype.data == "sdist"
and request.db.query(
request.db.query(File)
.filter((File.release == release) & (File.packagetype == "sdist"))
.exists()
).scalar()
):
raise _exc_with_message(
HTTPBadRequest, "Only one sdist may be uploaded per release."
)

Fixing this is probably blocked on #12245, since there is not such a thing as a normalized source distribution filename in every case until that is resolved.

@di di added blocked Issues we can't or shouldn't get to yet and removed requires triaging maintainers need to do initial inspection of issue labels Apr 19, 2024
@nanonano
Copy link
Author

Thanks for this, I didn't release the filename of the tar file had been changed to lowercase.

Now I need to figure out what has caused this change, the versions of build (1.2.1) and twine (5.0.0) are the same.

@di
Copy link
Member

di commented Apr 19, 2024

The change is caused by tools starting to support PEP 625. In your case, it is probably setuptools. In your case, bittytax-0.5.3.dev3.tar.gz is now the "right" filename for this, and nothing should change there.

Once #12245 is fixed, this should go away, and there's probably nothing you need to do here in the meantime.

@nanonano
Copy link
Author

OK great that explains it, thanks for your help.

I'll probably just up-version to get around the issue.

@james-rms james-rms mentioned this issue Apr 24, 2024
Merged
james-rms added a commit to foxglove/mcap that referenced this issue Apr 24, 2024
@james-rms
bump patch versions on all support packages to work around pypi/wareh…
50db2f4 
…ouse#15824
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocked Issues we can't or shouldn't get to yet bug 🐛
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@di @nanonano