Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix curl command error in URLDownloader.prefetch_filename() #939

Merged
merged 1 commit into from
May 15, 2024

Conversation

pyther
Copy link

@pyther pyther commented May 14, 2024

This reverts #626 which was added due to #624.

Previously, combining --head and --request GET allowed curl to perform a GET request and stop after receiving the headers. This workaround was added to handle servers that return HTTP Forbidden or incorrect headers for a HEAD request. However, with the update to curl 8.6.0 on macOS 14.5, this combination results in an error ("curl: (8) Weird server reply").

Previously, combining '--head' and '--request GET' allowed curl to perform a
GET request and stop after receiving the headers. This workaround was added to
handle servers that return HTTP Forbidden or incorrect headers for a HEAD
request. However, with the update to curl 8.6.0 on macOS 14.5, this
combination results in an error ("curl: (8) Weird server reply").
@homebysix homebysix self-assigned this May 14, 2024
@homebysix
Copy link
Member

For reference, this GET was added in PR #626 in response to issue #624.

I aim to test this change against the recipes mentioned in those two places and report back here. If you also have test results to contribute, please do. I'm especially interested in what this reversion will break, if anything.

@homebysix homebysix added the bug label May 15, 2024
@homebysix
Copy link
Member

‌openjdk8-binaries.download.recipe is used as an example of a recipe that was fixed by the addition of the GET. As of AutoPkg 2.7.2 and macOS 14.5, this recipe is now broken again.

% ./Code/autopkg run -vv ~/Downloads/openjdk8-binaries.download.recipe 
Processing ~/Downloads/openjdk8-binaries.download.recipe...
WARNING: ~/Downloads/openjdk8-binaries.download.recipe is missing trust info and FAIL_RECIPES_WITHOUT_TRUST_INFO is not set. Proceeding...
URLDownloader
{'Input': {'prefetch_filename': True,
           'url': 'https://api.adoptopenjdk.net/v3/binary/latest/8/ga/mac/x64/jdk/hotspot/normal/adoptopenjdk?project=jdk'}}
URLDownloader: No value supplied for CHECK_FILESIZE_ONLY, setting default value of: False
URLDownloader: ERROR: (8) Weird server reply

curl: (8) Weird server reply

Failed.
Receipt written to ~/Library/AutoPkg/Cache/com.github.autopkg.dwenger.download.openjdk8-binaries/receipts/openjdk8-binaries.download-receipt-20240514-202928.plist

The following recipes failed:
    ~/Downloads/openjdk8-binaries.download.recipe
        Error in com.github.autopkg.dwenger.download.openjdk8-binaries: Processor: URLDownloader: Error: curl: (8) Weird server reply

Nothing downloaded, packaged or imported.

Running the same recipe against the modified URLDownloader processor used in this PR shows that the recipe is working in macOS 14.5 despite the absence of the GET:

% ./Code/autopkg run -vv ~/Downloads/openjdk8-binaries.download.recipe
Processing ~/Downloads/openjdk8-binaries.download.recipe...
WARNING: ~/Downloads/openjdk8-binaries.download.recipe is missing trust info and FAIL_RECIPES_WITHOUT_TRUST_INFO is not set. Proceeding...
URLDownloader
{'Input': {'prefetch_filename': True,
           'url': 'https://api.adoptopenjdk.net/v3/binary/latest/8/ga/mac/x64/jdk/hotspot/normal/adoptopenjdk?project=jdk'}}
URLDownloader: No value supplied for CHECK_FILESIZE_ONLY, setting default value of: False
URLDownloader: Filename prefetched from the HTTP Content-Disposition header: OpenJDK8U-jdk_x64_mac_hotspot_8u412b08.tar.gz
URLDownloader: Storing new Last-Modified header: Fri, 19 Apr 2024 16:04:25 GMT
URLDownloader: Storing new ETag header: "0x8DC608A62533D4F"
URLDownloader: Downloaded ~/Library/AutoPkg/Cache/com.github.autopkg.dwenger.download.openjdk8-binaries/downloads/OpenJDK8U-jdk_x64_mac_hotspot_8u412b08.tar.gz
{'Output': {'download_changed': True,
            'etag': '"0x8DC608A62533D4F"',
            'last_modified': 'Fri, 19 Apr 2024 16:04:25 GMT',
            'pathname': '~/Library/AutoPkg/Cache/com.github.autopkg.dwenger.download.openjdk8-binaries/downloads/OpenJDK8U-jdk_x64_mac_hotspot_8u412b08.tar.gz',
            'url_downloader_summary_result': {'data': {'download_path': '~/Library/AutoPkg/Cache/com.github.autopkg.dwenger.download.openjdk8-binaries/downloads/OpenJDK8U-jdk_x64_mac_hotspot_8u412b08.tar.gz'},
                                              'summary_text': 'The following '
                                                              'new items were '
                                                              'downloaded:'}}}
EndOfCheckPhase
{'Input': {}}
{'Output': {}}
Receipt written to ~/Library/AutoPkg/Cache/com.github.autopkg.dwenger.download.openjdk8-binaries/receipts/openjdk8-binaries.download-receipt-20240514-203013.plist

The following new items were downloaded:
    Download Path                                                                                                                                      
    -------------                                                                                                                                      
    ~/Library/AutoPkg/Cache/com.github.autopkg.dwenger.download.openjdk8-binaries/downloads/OpenJDK8U-jdk_x64_mac_hotspot_8u412b08.tar.gz

I found 41 recipes in the AutoPkg org that use prefetch_filename, and I ran 38 of them before and after this URLDownloader change on macOS 14.5. I found 13 recipes failed with this PR, versus 36 failures on the master branch. That's good enough for me.

@homebysix homebysix self-requested a review May 15, 2024 04:39
@homebysix homebysix merged commit 9f4ea8b into autopkg:master May 15, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants