Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Find-PSResource -Name 'Az.Accounts' nuget v3 to get latest version of a module gets /recent.json instead of just getting /latest.json. #1603

Open
3 tasks done
o-l-a-v opened this issue Mar 15, 2024 · 3 comments
Open
3 tasks done

Find-PSResource -Name 'Az.Accounts' nuget v3 to get latest version of a module gets /recent.json instead of just getting /latest.json. #1603

o-l-a-v opened this issue Mar 15, 2024 · 3 comments
Labels
Area-Find Issue-Performance
Milestone
vNext

Comments

@o-l-a-v
Copy link
Contributor

o-l-a-v commented Mar 15, 2024 •
edited

Prerequisites

  • Write a descriptive title.
  • Make sure you are able to repro it on the latest released version
  • Search the existing issues.

Steps to reproduce

I'm experimenting with pwsh.gallery by @JustinGrote (hope it's OK :) ) + PSResourceGet.

Added it as repository like so:

Register-PSResourceRepository -Name 'pwsh.gallery' -Uri 'https://pwsh.gallery/index.json'

Then I wanted to see how fast it can return latest version of a given module. I did:

Find-PSResource -Repository 'pwsh.gallery' -Name 'Az.Accounts' -Verbose -Debug

I notice that PSResourceGet does three API calls where the first and last seems totally unneccessary for getting latest version.

One can determine latest version by just getting https://pwsh.gallery/az.accounts/index.json, or one could go straight to https://pwsh.gallery/az.accounts/page/latest.json.

PS > Find-PSResource -Repository 'pwsh.gallery' -Name 'Az.Accounts' -Verbose -Debug
DEBUG: In FindPSResource::ProcessResourceNameParameterSet()
DEBUG: Filtering package name(s) on wildcards
DEBUG: In FindHelper::FindByResourceName()
DEBUG: Parameters passed in >>> Name: 'Az.Accounts'; ResourceType: 'None'; VersionRange: ''; NuGetVersion: ''; VersionType: 'NoVersion'; Version: ''; Prerelease: 'False'; Tag: ''; Repository: 'pwsh.gallery'; IncludeDependencies 'False'
DEBUG: Searching through repository 'pwsh.gallery'
DEBUG: In FindHelper::SearchByNames()
DEBUG: No version specified, package name is specified
DEBUG: In V3ServerAPICalls::FindName()
DEBUG: In V3ServerAPICalls::FindNameHelper()
DEBUG: In V3ServerAPICalls::GetVersionedPackageEntriesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::GetResourcesFromServiceIndex()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/index.json'
DEBUG: In V3ServerAPICalls::FindRegistrationsBaseUrl()
DEBUG: In V3ServerAPICalls::GetVersionedResponsesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/index.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementsFromResponse()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/page/recent.json'
DEBUG: In V3ServerAPICalls::IsLatestVersionFirstForSearch()
DEBUG: 'Az.Accounts' version parsed as '2.16.0'
DEBUG: Found package 'Az.Accounts' version '2.16.0'
DEBUG: Package 'Az.Accounts' returned from server
DEBUG: Package 'Az.Accounts' was previously discovered and returned

Name        Version Prerelease Repository   Description
----        ------- ---------- ----------   -----------
Az.Accounts 2.16.0             pwsh.gallery

PS >

Questions:

  1. Wouldn't it be better to just get <nugetv3api>/az.accounts/index.json or <nugetv3api>/az.accounts/page/latest.json?
  2. Would it make sense to support -Version 'Latest' as input to Find-PSResource?

Expected behavior

When getting latest version only, Find-PSResource does just enough API calls, to 1) speed up end user experience, and 2) not put unneccessary load on the target repository/API.

Actual behavior

Find-PSResource gets <nugetv3api>/az.accounts/page/recent.json when getting latest version only.

Error details

No response

Environment data

PowerShell v7.4.1 x64 on Windows 11 23H2
Microsoft.PowerShell.PSResourceGet v1.0.3

Visuals

No response
@o-l-a-v
Copy link
Contributor Author

o-l-a-v commented Mar 15, 2024 •
edited

If asking for two packages PSResourceGet seems to do a total of six API calls.

PS > Find-PSResource -Repository 'pwsh.gallery' -Name 'Az.Accounts','Az.Resources' -Verbose -Debug
DEBUG: In FindPSResource::ProcessResourceNameParameterSet()
DEBUG: Filtering package name(s) on wildcards
DEBUG: In FindHelper::FindByResourceName()
DEBUG: Parameters passed in >>> Name: 'Az.Accounts,Az.Resources'; ResourceType: 'None'; VersionRange: ''; NuGetVersion: ''; VersionType: 'NoVersion'; Version: ''; Prerelease: 'False'; Tag: ''; Repository: 'pwsh.gallery'; IncludeDependencies 'False'
DEBUG: Searching through repository 'pwsh.gallery'
DEBUG: In FindHelper::SearchByNames()
DEBUG: No version specified, package name is specified
DEBUG: In V3ServerAPICalls::FindName()
DEBUG: In V3ServerAPICalls::FindNameHelper()
DEBUG: In V3ServerAPICalls::GetVersionedPackageEntriesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::GetResourcesFromServiceIndex()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/index.json'
DEBUG: In V3ServerAPICalls::FindRegistrationsBaseUrl()
DEBUG: In V3ServerAPICalls::GetVersionedResponsesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/index.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementsFromResponse()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.accounts/page/recent.json'
DEBUG: In V3ServerAPICalls::IsLatestVersionFirstForSearch()
DEBUG: 'Az.Accounts' version parsed as '2.16.0'
DEBUG: Found package 'Az.Accounts' version '2.16.0'
DEBUG: Package 'Az.Accounts' returned from server
DEBUG: Package 'Az.Accounts' was previously discovered and returned

DEBUG: No version specified, package name is specified
DEBUG: In V3ServerAPICalls::FindName()
DEBUG: In V3ServerAPICalls::FindNameHelper()
DEBUG: In V3ServerAPICalls::GetVersionedPackageEntriesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::GetResourcesFromServiceIndex()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/index.json'
DEBUG: In V3ServerAPICalls::FindRegistrationsBaseUrl()
DEBUG: In V3ServerAPICalls::GetVersionedResponsesFromRegistrationsResource()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.resources/index.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementsFromResponse()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromItemsElement()
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.resources/page/recent.json'
DEBUG: In V3ServerAPICalls::GetMetadataElementFromIdLinkElement()
DEBUG: In V3ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://pwsh.gallery/az.resources/page/older.json'
DEBUG: In V3ServerAPICalls::IsLatestVersionFirstForSearch()
DEBUG: 'Az.Resources' version parsed as '7.0.0-preview'
DEBUG: 'Az.Resources' version parsed as '6.16.0'
DEBUG: Found package 'Az.Resources' version '6.16.0'
DEBUG: Package 'Az.Resources' returned from server
DEBUG: Package 'Az.Resources' was previously discovered and returned
Name         Version Prerelease Repository   Description
----         ------- ---------- ----------   -----------
Az.Accounts  2.16.0             pwsh.gallery
Az.Resources 6.16.0             pwsh.gallery

PS >

PSResourceGet seems to be better optimized with nuget v2 / regular PowerShell Gallery:

PS > Find-PSResource -Repository 'PSGallery' -Name 'Az.Accounts','Az.Resources' -Verbose -Debug
DEBUG: In FindPSResource::ProcessResourceNameParameterSet()
DEBUG: Filtering package name(s) on wildcards
DEBUG: In FindHelper::FindByResourceName()
DEBUG: Parameters passed in >>> Name: 'Az.Accounts,Az.Resources'; ResourceType: 'None'; VersionRange: ''; NuGetVersion: ''; VersionType: 'NoVersion'; Version: ''; Prerelease: 'False'; Tag: ''; Repository: 'PSGallery'; IncludeDependencies 'False'
DEBUG: Searching through repository 'PSGallery'
DEBUG: In FindHelper::SearchByNames()
DEBUG: No version specified, package name is specified
DEBUG: In V2ServerAPICalls::FindName()
DEBUG: In V2ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'&$inlinecount=allpages&$filter=IsLatestVersion and Id eq 'Az.Accounts''
DEBUG: Found package 'Az.Accounts' version '2.16.0'
DEBUG: Package 'Az.Accounts' returned from server
DEBUG: Package 'Az.Accounts' was previously discovered and returned

DEBUG: No version specified, package name is specified
DEBUG: In V2ServerAPICalls::FindName()
DEBUG: In V2ServerAPICalls::HttpRequestCall()
DEBUG: Request url is 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Resources'&$inlinecount=allpages&$filter=IsLatestVersion and Id eq 'Az.Resources''
DEBUG: Found package 'Az.Resources' version '6.16.0'
DEBUG: Package 'Az.Resources' returned from server
DEBUG: Package 'Az.Resources' was previously discovered and returned
Name         Version Prerelease Repository Description
----         ------- ---------- ---------- -----------
Az.Accounts  2.16.0             PSGallery  Microsoft Azure PowerShell - Accounts credential management cmdlets for Azure Resource Manager in Windows PowerShell and PowerShell Core.…
Az.Resources 6.16.0             PSGallery  Microsoft Azure PowerShell - Azure Resource Manager and Active Directory cmdlets in Windows PowerShell and PowerShell Core.  Manages subscriptions, tenants, resource groups, deployment templates, providers,…

PS >

@o-l-a-v
Copy link
Contributor Author

o-l-a-v commented Mar 31, 2024

Other nuget v3 API's than pwsh.gallery seems to act different. Take nuget.org for instance. It does not have <nugetv3api>/<package>/page/latest.json.

Example: NuGet.Versioning:

So maybe this isn't feasible with nuget v3 APIs after all.

@JustinGrote
Copy link
Contributor

@o-l-a-v the v3 protocol specifically states you have to first look up the registration (which you can cache), then use the registration to find the package index, then use the package index to find the catalog leaf where your desired version resides.

If you know the version already, you can make a shortcut there, the protocol allows that.

Also, "recent.json" is a pwsh.gallery nomenclature. pwsh.gallery is optimized for the fact that the majority of powershell installations want the latest version and prerelease version, so to minimize bandwidth recent.json is generated automatically with only the dependency metadata included to minimize data transfer over the wire and serialization delay, but nuget v3 is designed with static fetching in mind.

From what I see the fetch is mostly working to spec for a cold fetch, though I don't think it caches or uses ETag today to speed up future queries. There's one call to recent.json that isn't necessary, since the index.json would show the version being searched for is in older.json

@SydneyhSmith SydneyhSmith added this to the vNext milestone Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area-Find Issue-Performance
Projects
None yet
Milestone
vNext
Development

No branches or pull requests
4 participants
@o-l-a-v @JustinGrote @anamnavi @SydneyhSmith