Created SearchCommandSpec.md #9718

advay26-zz · 2020-06-25T20:11:00Z

Created a design spec for the Search feature I'm implementing for NuGet.exe. [Issue 9704]

dnfgituser · 2020-06-25T20:11:14Z

All CLA requirements met.

zkat

This is a great first draft! Let's iterate on it a bit :)

I'd love to get feedback from others on the team, too.

zkat · 2020-06-25T20:18:05Z

designs/SearchCommandSpec.md

+
+--------------------
+> Serilog.Extensions.Logging | v3.0.2-dev-10280 | DLs: 43115462
+-- More  --


You can probably cut this example down to 2-3 entries instead of the full actual results. This takes up a lot of space!

I've made this change in my latest commit.

zkat · 2020-06-25T20:27:15Z

designs/SearchCommandSpec.md

+
+## Who are the customers
+
+All NuGet.exe and .NET CLI users.


Those who want to search without going to the nuget gallery or using VS PMUI, maybe?

I've added this to the 'Who are the customers' section.

zkat · 2020-06-25T20:28:46Z

designs/SearchCommandSpec.md

+
+## Future Work
+
+* Enabling cross-platform functionality for the search feature


I think this is a core feature. It's not hard to add multi-platform pager support. We can work on this together.

Which functionality are we talking about? dotnet.exe implies cross-plat.

We have tabled the idea for the pager for now, and also decided to leave dotnet.exe functionality to future work

zkat · 2020-06-25T20:30:20Z

designs/SearchCommandSpec.md

+## Future Work
+
+* Enabling cross-platform functionality for the search feature
+* Adding optional arguments that allow users to customize their search queries


There are some optional arguments that we should definitely include in the current iteration. This spec should propose a few (I don't think we need to deal with skip/take for now, though, but being able to control prerelease and package type would be good, and not hard to add right out the door

Also, what about a source parameter to use a different NuGet server?

Agreed on source, this aligns with many of the other nuget commands.

Perhaps having the proposed usage text for both .net cli and nuget.exe is the best way to communicate that? it's succinct and well-understood. just an idea, it should be communicated somehow

I have added an 'Optional Arguments' section in 'Solution' that introduces Pre-Release, Package Type, Source, Take and Help as arguments available to users.

zkat · 2020-06-25T20:32:09Z

designs/SearchCommandSpec.md

+
+Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.
+```
+PS C:\> ./NuGet.exe search logging


How is the experience different for dotnet users? The way parameters are passed in to both CLIs is different, for example.

Should we, though, do dotnet support as "future work"?

In the latest commit, I have put down creating a prototype for dotnet.exe as a goal, and integrating Search into dotnet.exe as something for 'Future Work'

loic-sharma · 2020-06-25T21:11:27Z

designs/SearchCommandSpec.md

+
+## Who are the customers
+
+All NuGet.exe and .NET CLI users.


Why do we want to support NuGet.exe customers? Is nuget.exe list not enough? It may be helpful to describe the benefits and differences between list and search

I have attempted to address this in the 'Who are the customers' section. Let me know where I've fallen short in my explanation and I'd be happy to amend it further!

designs/SearchCommandSpec.md

loic-sharma · 2020-06-25T21:15:29Z

designs/SearchCommandSpec.md

+Commonly Used Types:
+Microsoft.Extensions.Logging.ILogger
+Microsoft.Extensions.Logging.ILoggerFactory
+Microsoft.Extensions.Logging.ILogger&lt;TCategoryName&gt;


Should we HTML decode the description?

This should be done by the API unless the description is double-encoded. Ideally the client does minimal manipulation on the returned description

loic-sharma · 2020-06-25T21:20:04Z

designs/SearchCommandSpec.md

+
+### 1. Combining the results from different sources
+
+Initially, we considered consolidating all search results from the various sources and displaying them together, rather than separating them based on their source. However, users may wish to know where a particular package is located. Also, combining the search results for different sources leads to the complex problem of how to combine these results most effectively.


I definitely agree with this section. That said, NuGet normally uses your nuget.config file to discover package sources to read from. Users may be confused since this search experience won't do that. Is this something we want to address somehow?

Where will it get its sources, then?

I took this to mean the sources still come from config, but they are not interleaved.

If there is one "section" per source, how will they be sorted? lex by name? lex by URL? same order as config?

This section was meant to convey that instead of interleaving the results from the various sources in nuget.config, we display the results for each source as a separate section, followed by the results for the next source, and so on. As far as I'm aware, the NuGet API's Search only returns results sorted by relevance. I have also added a -Source optional argument in the 'Optional Arguments' section in 'Solution' if a user wants to specify a source for the Search to target.

donnie-msft

😎 going to be awesome!

donnie-msft · 2020-06-25T21:38:02Z

designs/SearchCommandSpec.md

+
+## Solution
+
+Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.


Suggested change

Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.

The search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.

donnie-msft · 2020-06-25T21:39:18Z

designs/SearchCommandSpec.md

+
+## Solution
+
+Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.


What if I want to search a specific source, can we have an option for -source to specify one? (eg, I don't want to see a bajillion "logger" results from nuget.org, but I want to browse my org's internal ones.

I have added this in the 'Optional Arguments' section at the end of 'Solution'.

donnie-msft · 2020-06-25T21:42:53Z

designs/SearchCommandSpec.md

+PS C:\> ./NuGet.exe search logging
+```
+
+Each search result displays the name, version, number of downloads, and description of the package. Additionally, Search uses pagination to provide a clean viewing experience for users as they look at the results of their search. This ensures that users have access to the search results in order of decreasing relevance. A single page of results is shown at a time, and the user can look at the next page by pressing the __space__ key.


Can sort be specified? eg, -sort downloads, or -sort name?

What's the default sort? Should it be relevancy, download count, or name?
I think some API's are Server dependent (either registration or search), so depending on which you use, it might be nice to tell the user what order they're about to see, and if they can change that.

NuGet search protocol does not support sorting today. I'd suggest keeping this out of the MVP

The only sort provided by our official API is relevance. /cc @t-mog-msft who is plumbing some of this through. downloads and name are not supported by API and there is no concrete plan for this yet.

So the client won't be sorting, gotcha.
I'd still like to see some note to the user, like (sorted by relevance according to the source) ?
We get so many questions about lack of sort/filter, so somehow stating it might be valuable. At least consider it for -verbosity detailed.

donnie-msft · 2020-06-25T21:44:58Z

designs/SearchCommandSpec.md

+PS C:\> ./NuGet.exe search logging
+```
+
+Each search result displays the name, version, number of downloads, and description of the package. Additionally, Search uses pagination to provide a clean viewing experience for users as they look at the results of their search. This ensures that users have access to the search results in order of decreasing relevance. A single page of results is shown at a time, and the user can look at the next page by pressing the __space__ key.


What about other tidbits like deprecation, renamed packages, or known vulnerabilities?

I haven't added these pieces of information yet, but in 'Open Questions', I've addressed the fact that we may want to consider additional/alternative information for the packages returned by Search.

donnie-msft · 2020-06-25T21:46:59Z

designs/SearchCommandSpec.md

+
+### 1. Combining the results from different sources
+
+Initially, we considered consolidating all search results from the various sources and displaying them together, rather than separating them based on their source. However, users may wish to know where a particular package is located. Also, combining the search results for different sources leads to the complex problem of how to combine these results most effectively.


Where will it get its sources, then?

donnie-msft · 2020-06-25T21:47:54Z

designs/SearchCommandSpec.md

+
+## Future Work
+
+* Enabling cross-platform functionality for the search feature


Which functionality are we talking about? dotnet.exe implies cross-plat.

joelverhagen · 2020-06-25T22:00:53Z

designs/SearchCommandSpec.md

+
+### 2. Displaying results by printing the output stream to the console without pagination
+
+If we were to simply print the output to the console as a regular stream, the most visible results would be the ones at the end of the output stream, since these would be located right above the next command prompt in the terminal. Pagination, as proposed in this document, ensures that the most relevant results are shown to the users first.


How will the pager be implemented? IIRC git uses less by default but it is configurable. Writing our own might be a little tricky right? you have to figure out terminal dimensions maybe?

My plan is to use more by default on Windows (it's built into cmd.exe /c more). This is what the current prototype is using. We would use less by default on *nix systems. Additionally, I think as a stretch goal, we should support the standard-ish $PAGER env var.

For now I have put the pager idea in 'Future Work' and we can consider building it in later if we have a consensus.

designs/SearchCommandSpec.md

zivkan · 2020-06-25T22:03:30Z

designs/SearchCommandSpec.md

+
+### 2. Displaying results by printing the output stream to the console without pagination
+
+If we were to simply print the output to the console as a regular stream, the most visible results would be the ones at the end of the output stream, since these would be located right above the next command prompt in the terminal. Pagination, as proposed in this document, ensures that the most relevant results are shown to the users first.


As someone who is somewhat used to using Linux, I don't want the tool itself to do pagination, I'll pipe the output to more or less myself. Was the proposal to do this within nuget because of customer feedback about existing features, or trying to be pro-active?

As a developer on NuGet, I'm worried about having to detect when the command is interactive or being piped to a text file, or being caputed by another process, and therefore should not pause for user-input.

I agree, it adds complexity and should only be part of the MVP if it's really the goal. it feels very different from other nuget commands. Do other .NET CLI commands have pager behavior on by default?

What's the real drawback from a customer experience perspective to making pagination the default behavior? I understand that it might be inconsistent with other elements of the dotnet CLI, but consistency is only a priority when it serves to improve the customer experience, which I don't think it does here.

I'm not sure many people will have the knowledge or motivation to pipe the output for pagination themselves, I wouldn't. The default experience without pagination ends the execution of the list/search command with the least relevant results showing. In the case of there being many results, scrolling to the top for the most relevant results could be a chore and scrolling too quickly could cause you to easily miss/pass the desired package.

Another potential solution is to list results in reverse relevance order, but my concern would be that that would be difficult to communicate clearly and inherently counterintuitive.

The ultimate difference will be that with the current behavior, a user probably has to look for their desired package among the results. With the proposed behavior, it's much more likely they'll just see it immediately with most relevant packages being surfaced in the first "page."

As a developer on NuGet, I'm worried about having to detect when the command is interactive or being piped to a text file, or being caputed by another process, and therefore should not pause for user-input.

I'm not sure what you mean by "having to detect" here. Could you elaborate?

What's the real drawback from a customer experience perspective to making pagination the default behavior?

Shouldn't we be justifying why to spend effort on something, rather than why not to?

The feature is already built into every operating system or shell. What's being proposed (space to scroll 1 screen) has less features than what's built-in (space to scroll 1 screen, or enter to scroll 1 line on Windows. Linux's more also has text search built-in). We'd just be spending development resources duplicating features that already exist and are already available for all customers, using a technique that works for all programs that simply output text to the console. Composing excellent tools together often has a better outcome than a single development team trying to re-invent the most basic features of everyone else's wheels into one integrated product. But this is probably being proposed to help people who are not like me.

I don't know about Windows or Mac, but on Linux I can use dir | less to get scroll up and well as down, either one line at a time, or one screen at a time, plus text search forwards or backwards, and a whole lot of other features. I can use dir | grep ..., dir | sed ... to do more advanced scripting. Learning how to dir | more opens the door to learning about all these scripting techniques. Building it into an app, and therefore hiding away the entire scripting world from a beginner harms their education, even if it's marginally easier to use. And my opinion is that | more is trivial to learn, so the benefit is really marginal.

I was going to write that if we want to invest in a command line search that is interactive, we should consider the command line UI library that Loic used recently and build something much more unique (although Loic showed me that something like this already exists for npm at least). But then when I found Loic's app, it's already a command line NuGet search tool.

I'm not sure many people will have the knowledge or motivation to pipe the output for pagination themselves,

I disagree. My hypothesis is that anyone who has spent much time in any command line will have learnt about piping output to more, since this works for all command line apps on all operating systems. Any web search about how to scroll one screen at a time on the command line shows many blog posts, stack overflow questions & answers, and other resources that introduce | more.

I'm not sure what you mean by "having to detect" here. Could you elaborate?

With this proposal if I run nuget search whatever, NuGet is going to print one screen's worth of text, tell me to press space to continue, then stop and wait for input before outputting more. If I run nuget search whatever > results.txt, the text telling me to press space will not be output to the terminal, so NuGet just hangs waiting for input that will never come because I don't know it's waiting for me to press anything. What about if I run nuget search term1 | grep -i term2, is that going to hang waiting for input?

Since we don't currently have a consensus on the pager idea, I have put it in 'Future Work' for now.

designs/SearchCommandSpec.md

joelverhagen · 2020-06-25T22:29:19Z

designs/SearchCommandSpec.md

+
+--------------------
+> Microsoft.Extensions.Logging.Debug | v5.0.0-preview.6.20305.6 | DLs: 58643995
+Debug output logger provider implementation for Microsoft.Extensions.Logging. This logger logs messages to a debugger monitor by writing messages with System.Diagnostics.Debug.WriteLine().


Is there a limit of the description shown? IIRC we support up to 4000 characters on nuget.org but there is no limit from the protocol

I have built in a character limit in my latest commit. The description is truncated if it exceeds a certain character limit. You can see an example of this in the first result in the 'Solution' section.

Consider full/more characters when using -verbosity detailed.

joelverhagen · 2020-06-25T22:31:12Z

designs/SearchCommandSpec.md

+When using NuGet 3.x this package requires at least version 3.4.
+
+--------------------
+> Microsoft.Extensions.Logging.Console | v5.0.0-preview.6.20305.6 | DLs: 61796431


Will this show normalized version or full version (includes build metadata)? API returns full version.

Looking at some specific examples, like NuGet.Protocol, the API seems to return the normalized version already, but feel free to correct me if I'm wrong.

The NuGet.Protocol API returns whatever the HTTP endpoint returns. Per protocol specification, this should be the full version:
https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource#search-result

I don't know the specific API you are using in client, but if a NuGetVersion is involved instead of a raw string, you can choose via one of these two methods:
https://github.com/NuGet/NuGet.Client/blob/088004379cb99890f766e9b7a0a29f7b080f888b/src/NuGet.Core/NuGet.Versioning/SemanticVersionBase.cs#L13-L30

Ok, I'll make to sure to incorporate this into how we display results.

joelverhagen · 2020-06-25T23:08:34Z

designs/SearchCommandSpec.md

+Each search result displays the name, version, number of downloads, and description of the package. Additionally, Search uses pagination to provide a clean viewing experience for users as they look at the results of their search. This ensures that users have access to the search results in order of decreasing relevance. A single page of results is shown at a time, and the user can look at the next page by pressing the __space__ key.
+
+```
+====================


What will the behavior be if there are no results? Some text?

The 'Solution' section now includes an example of what is shown if a particular source returns no results.

joelverhagen · 2020-06-25T23:24:56Z

designs/SearchCommandSpec.md

+When using NuGet 3.x this package requires at least version 3.4.
+
+--------------------
+> Microsoft.IdentityModel.Logging | v6.6.0 | DLs: 120177247


Just a question, is it normal that we show the v in front of the package version? Do we do that elsewhere? I'm used to just seeing the version without the v prefix but maybe I am totally wrong

I've included this change in the latest commit.

nkolev92

What the options this command can take?

See: https://docs.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-list for inspiration.

IncludeDelisted is not relevant there.

designs/SearchCommandSpec.md

nkolev92 · 2020-06-26T02:56:01Z

designs/SearchCommandSpec.md

+
+Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.
+```
+PS C:\> ./NuGet.exe search logging


The one thing never mentioned is the nuget.exe list command.

The list command is really the old V2 "search".
That command already exists and it has a certain set of options and parameters.

We might want to deprecate list first or at least in the same timeframe (potentially @dominoFire can resurrect his PR).
NuGet/NuGet.Client#2772
NuGet/NuGet.Client#2772

I have addressed the comparisons between list and search in 'Who are the customers' in the latest commit, but if this section is incomplete or incorrect, please let me know!

Only thing I'd suggest is calling out the list deprecation plan.

We want to write a feature that people will use. Deprecating list and pointing to search will be something that should increase the usage of the command.

designs/SearchCommandSpec.md

joelverhagen · 2020-06-29T17:36:59Z

designs/SearchCommandSpec.md

+| Name | Description | Usage |
+| ---  |     ---     |  :-:  |
+| PreRelease | `true` or `false` determining whether to include pre-release packages | -PreRelease `<true/false>` |
+| PackageTypes | The package type to use to filter packages. If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|


Adoption of package type filtering on the server side is likely very sparse. AFAIK the only implementor is nuget.org. Other feeds like Azure DevOps and GitHub probably don't support it today. Therefore, we should be careful about the wording and graceful failures. There is a way to check if a server supports package type filtering so I would recommend spec'ing here what the behavior is if the server does not support package type filtering. It's up to you, but maybe say "Package type filtering is not supported on this package source." for each source not support it instead of showing any search results at all.

BTW, you detect package type filtering capabilities via the SearchQueryService/3.5.0 type in the service index, e.g. just search for that string here https://api.nuget.org/v3/index.json and you'll see.

Suggested change

| PackageTypes | The package type to use to filter packages. If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|

| PackageTypes | The package type to use to filter packages (if supported by the source). If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|

Should indicate whether the Filter actually got applied.
I think in my ideal world, we wouldn't simply return empty, but instead output:
(at least consider for -verbosity detailed)

"Applying Filters"

"Ignoring Filters (not supported by source)"

Error: "Invalid Filter"

I've added Donnie's disclaimer in the PackageTypes description, and -Verbosity detailed can provide additional information on the filters being used and whether they were applied or not.

joelverhagen · 2020-06-29T17:37:27Z

designs/SearchCommandSpec.md

+| PreRelease | `true` or `false` determining whether to include pre-release packages | -PreRelease `<true/false>` |
+| PackageTypes | The package type to use to filter packages. If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|
+| Source | Specific package source(s) to search instead of querying the default sources in __nuget.config__ | -Source `<Source URL>`|
+| Take | The number of results to return | -Take `<positive integer>` |


nit: Perhaps specify the default take in the description?

+1 to specifying the default.

joelverhagen · 2020-06-29T17:38:41Z

designs/SearchCommandSpec.md

+
+| Name | Description | Usage |
+| ---  |     ---     |  :-:  |
+| PreRelease | `true` or `false` determining whether to include pre-release packages | -PreRelease `<true/false>` |


nit: other commands like install and list have prerelease off by default and included with just a switch like -Prerelease. AFAIK you don't pass in true or false but I may be wrong here. (I'm just looking at nuget.exe list -help and nuget.exe install -help)

You may find some option flags requiring true/false in nuget.exe, which is a bug (so don't copy everything you see there 😄).
Maybe even rename this to IncludePreRelease ? Implying that the default is false.

I've changed -PreRelease to just be a switch. I haven't renamed it to -IncludePreRelease simply to maintain consistency with other nuget commands like list.

joelverhagen · 2020-06-29T17:43:43Z

designs/SearchCommandSpec.md

+
+## Solution
+
+The search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.


Consider also mentioned whether the query term is required (nuget.exe list does not require one but it's up to you), how to specify multiple terms (in quotes, most likely), and the query syntax (determined by server implementation, e.g. nuget.org has this.

joelverhagen · 2020-06-29T17:45:35Z

designs/SearchCommandSpec.md

+====================
+Source: NuGet.org
+--------------------
+> Microsoft.Extensions.Logging.Abstractions | 5.0.0-preview.6.20305.6 | Downloads: 345,145,935


I really like this lines format. It lends itself to grep'ing very well since there is a well-known character > at the beginning of the line. Consider indenting the description lines by at least one space so that a description containing > is not a false positive. Also, the indenting might look nice a la dotnet list package

I've included this change, but the PrintJustified method I used in my prototype needs some adapting to fit my needs here so I'm going to continue working on this.

joelverhagen · 2020-06-29T18:06:06Z

designs/SearchCommandSpec.md

+| PackageTypes | The package type to use to filter packages. If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|
+| Source | Specific package source(s) to search instead of querying the default sources in __nuget.config__ | -Source `<Source URL>`|
+| Take | The number of results to return | -Take `<positive integer>` |
+| Help | Displays help information for the command | -Help |


Riffing off what @donnie-msft said, what about -Verbosity here?

When you increase verbosity on nuget.exe list, it also shows the API URL that is being hit. Might be nice, a la #9630 (shameless plug)

donnie-msft

Left a few more comments, but looking good!

loic-sharma · 2020-06-29T18:23:05Z

designs/SearchCommandSpec.md

+
+## Who are the customers
+
+This targets all NuGet.exe and .NET users that want to be able to search for packages from the command line, rather than using NuGet Gallery or VS PMUI. Also, while `nuget.exe list` allows users to search different package sources for packages with a search query, the results are returned sorted by the package Id, rather than by relevance, as proposed for this Search command. Further, the Search command would provide more information on the packages than List currently does.


This is great! Another common problem that is addressed by your spec are customers that use nuget.exe on a private NuGet server that only supports the NuGet V3 protocol but not V2 (in case you're interested, see loic-sharma/BaGet#187). Today, these customers will get the following error if they use the nuget list command:

This version of nuget.exe does not support listing packages from package source 'http://get.xxxxxxxx.com/v3/index.json'.

Your spec gives these customers an escape hatch if the server supports the search protocol. We could improve the error message to say something like:

This version of nuget.exe does not support listing packages from package source 'http://get.xxxxxxxx.com/v3/index.json'. Consider using the nuget.exe search command instead.

What do you think?

Primarily, I added the verbosity optional argument, and added the indentation for the description (although the PrintJustified method that I used in my prototype needs some adapting to fit my need here).

joelverhagen · 2020-07-01T17:09:26Z

designs/SearchCommandSpec.md

+> Microsoft.Extensions.Logging.Abstractions | 5.0.0-preview.6.20305.6 | Downloads: 345,145,935
+  Logging abstractions for Microsoft.Extensions.Logging.
+
+Commonly Used Types:


nit: wouldn't this be indented too?

Yes, that's the aim. There was a bug in PrintJustified, which is the Console Method that prints indented text to the terminal for commands like list. I fixed that and created another PR for it , so once that's approved the indentation should work fine. I'll fix it in the spec in the meanwhile.

joelverhagen · 2020-07-01T17:10:41Z

designs/SearchCommandSpec.md

+
+| Name | Description | Usage |
+| ---  |     ---     |  :-:  |
+| PackageTypes | The package type to use to filter packages (if supported by the source). If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|


nit, maybe a better user experience is saying "package type filtering is not supported on this feed" instead of the same message that occurs if they have too specific of parameters and no results are found. Said another way, this problem of no package type filtering cannot be resolved by the user but no results probably can.

I think we should drop PackageTypes from the initial implementation.

It increases the scope of the feature.

We don't have super well understood use cases for package types yet. dotnet tool search is an obvious example but that has it's use right now.

I'd prefer to add that at a later point based on community feedback.

joelverhagen · 2020-07-01T17:11:33Z

designs/SearchCommandSpec.md

+
+* _quiet_ - Package ID, Version
+* _normal_ - Package ID, Version, Downloads, Preview of Description
+* _detailed_ - Package ID, Version, Downloads, Full Description, Other information regarding the arguments and filters applied to the search


nit: can the query URLs be printed out at this verbosity level too? Not a big deal but it can help in some cases to diagnose problems.

chrisraygill · 2020-07-01T20:35:19Z

designs/SearchCommandSpec.md

+## Goals
+
+* Providing search functionality for NuGet packages through the commandline for NuGet.exe
+* Implementing a prototype of this feature for dotnet.exe


What does a prototype entail here? Are there areas where the experience between nuget.exe and dotnet.exe would diverge?

So far my focus has been on the NuGet.exe experience, and I haven't been able to explore what the dotnet.exe experience would look like. Maybe @zkat could answer this better?

chrisraygill · 2020-07-01T20:37:40Z

designs/SearchCommandSpec.md

+
+```
+====================
+Source: NuGet.org


How are we deciding the display order of the sources? Response order, alphabetical, # of results?

Unless specific sources are specified, the sources are queried in the order they appear in the nuget.config file (as far as I know).

chrisraygill · 2020-07-01T20:39:59Z

designs/SearchCommandSpec.md

+| Name | Description | Usage |
+| ---  |     ---     |  :-:  |
+| PackageTypes | The package type to use to filter packages (if supported by the source). If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|
+| PreRelease | Pre-release packages are not included by default, but can be included by using this argument | -PreRelease |


This is a nitpick, but I think options might typically be lowercased https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet. I'm not sure if the casing here may confuse anyone.

Other NuGet commands like list seem to use uppercase options similar to the ones used here, so I thought it made sense to do this in the interest of consistency

Oh that's interesting. It makes sense to be consistent with nuget.exe docs since that's what we're focusing on 👍

nuget.exe uses powershell -PascalCase style arguments, whereas dotnet uses --posix or --snake-case style arguments.

chrisraygill · 2020-07-01T20:42:54Z

designs/SearchCommandSpec.md

+
+### 1. Combining the results from different sources
+
+Initially, we considered consolidating all search results from the various sources and interleaving them together, rather than separating them based on their source. However, users may wish to know where a particular package is located. Also, combining the search results for different sources leads to the complex problem of how to combine these results most effectively.


Don't we combine results in Visual Studio? I imagine we should be able to use the same method to combine results here. My instinct is that I like has results from each source displayed separately, but I wonder how often customers may be using some configuration their team set up and actually aren't aware what source a particular package they need is coming from. @loic-sharma any input here?

We are using Lucene.NET.dll to combine the search results.
We cannot use Lucene.NET in dotnet.exe because of source-build concerns. See https://github.com/dotnet/source-build#net-core-source-build.

From their repo:

The key goal of this repository is to satisfy the official packaging rules of commonly used Linux distributions, such as Fedora and Debian. Many Linux distributions have similar rules. These rules tend to have two main principles: consistent reproducibility, and source code for everything.

We basically would need to implement client side merging with .NET runtime only dependencies.
Not sure if we have anything readily available.

Also merging search results falls apart when one feed is very slow and another one is fast on the command line. For the VSUI you can adjust the ordering dynamically as slow results come in. For a command line once you write a search result it can't be (easily) undone so you need to wait for all search pages to return before showing anything. This can lead to a subpar experience when you have a slow feed in the mix.

nkolev92

Looks great.
A few logistics comments.

I think we should not add PackageTypes support in V1. Beyond that fix the title of the spec maybe? Given that dotnet.exe is not part of V1.

I do have some concerns about shipping new features in nuget.ee but not in dotnet.exe but if PMs are fine, I won't insist.

nkolev92 · 2020-07-02T01:28:42Z

designs/SearchCommandSpec.md

+
+| Name | Description | Usage |
+| ---  |     ---     |  :-:  |
+| PackageTypes | The package type to use to filter packages (if supported by the source). If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. | -PackageType `<Package Type>`|


I think we should drop PackageTypes from the initial implementation.

It increases the scope of the feature.

We don't have super well understood use cases for package types yet. dotnet tool search is an obvious example but that has it's use right now.

I'd prefer to add that at a later point based on community feedback.

nkolev92 · 2020-07-02T01:31:25Z

designs/SearchCommandSpec.md

+* Adding further optional arguments that allow users to customize their search queries
+* Using columns rather than `|` to display results
+* Consider using pagination to display the search results
+* Allow multiple/no query terms


Deprecating list maybe?
Should list point to the new search command? Was that something we considered?

nkolev92 · 2020-07-02T01:33:47Z

designs/SearchCommandSpec.md

+
+Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.
+```
+PS C:\> ./NuGet.exe search logging


Only thing I'd suggest is calling out the list deprecation plan.

We want to write a feature that people will use. Deprecating list and pointing to search will be something that should increase the usage of the command.

Removed the -PackageTypes argument, alluded to potential deprecation of list, and made NuGet.exe rather than dotnet.exe the primary focus of this feature

nkolev92

Well done! 💯

Created SearchCommandSpec.md

19b53b6

zkat reviewed Jun 25, 2020

View reviewed changes

loic-sharma reviewed Jun 25, 2020

View reviewed changes

designs/SearchCommandSpec.md Show resolved Hide resolved

loic-sharma reviewed Jun 25, 2020

View reviewed changes

designs/SearchCommandSpec.md Outdated Show resolved Hide resolved

loic-sharma reviewed Jun 25, 2020

View reviewed changes

donnie-msft reviewed Jun 25, 2020

View reviewed changes

joelverhagen reviewed Jun 25, 2020

View reviewed changes

zivkan reviewed Jun 25, 2020

View reviewed changes

joelverhagen reviewed Jun 25, 2020

View reviewed changes

nkolev92 reviewed Jun 26, 2020

View reviewed changes

loic-sharma mentioned this pull request Jun 28, 2020

nuget list command failed loic-sharma/BaGet#187

Closed

Revised SearchCommandSpec in response to feedback from PR

6600781

advay26-zz requested review from zkat, joelverhagen, loic-sharma, nkolev92, zivkan, chrisraygill and donnie-msft June 29, 2020 17:24

joelverhagen reviewed Jun 29, 2020

View reviewed changes

donnie-msft approved these changes Jun 29, 2020

View reviewed changes

loic-sharma reviewed Jun 29, 2020

View reviewed changes

advay26-zz added 3 commits June 30, 2020 12:58

More revisions to the spec.

ef2f546

Primarily, I added the verbosity optional argument, and added the indentation for the description (although the PrintJustified method that I used in my prototype needs some adapting to fit my need here).

Added more information to -Verbosity detailed

a9899c7

Added more query options as 'Future Work'

9fc9470

advay26-zz requested review from joelverhagen and donnie-msft June 30, 2020 17:10

joelverhagen reviewed Jul 1, 2020

View reviewed changes

joelverhagen approved these changes Jul 1, 2020

View reviewed changes

advay26-zz added 2 commits July 1, 2020 13:39

Fixed indentation in example output

acbe82c

Fixed formatting

c9cbb1d

donnie-msft approved these changes Jul 1, 2020

View reviewed changes

loic-sharma approved these changes Jul 1, 2020

View reviewed changes

chrisraygill approved these changes Jul 1, 2020

View reviewed changes

nkolev92 reviewed Jul 2, 2020

View reviewed changes

Updated the Options, made NuGet.exe the focus

86f2733

Removed the -PackageTypes argument, alluded to potential deprecation of list, and made NuGet.exe rather than dotnet.exe the primary focus of this feature

advay26-zz requested a review from nkolev92 July 2, 2020 17:50

nkolev92 approved these changes Jul 2, 2020

View reviewed changes

advay26-zz merged commit 79af9ef into NuGet:dev Jul 2, 2020

advay26-zz deleted the dev-advay26-searchspec branch July 2, 2020 23:57


		## Future Work

		* Enabling cross-platform functionality for the search feature


		### 1. Combining the results from different sources

		Initially, we considered consolidating all search results from the various sources and displaying them together, rather than separating them based on their source. However, users may wish to know where a particular package is located. Also, combining the search results for different sources leads to the complex problem of how to combine these results most effectively.


		## Solution

		Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.

	Th search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.
	The search feature leverages the search service provided by the NuGet API. Search targets multiple sources and displays all of their results one after the other. The command line interface is simple: __./NuGet.exe search \<query\>__.


		### 2. Displaying results by printing the output stream to the console without pagination

		If we were to simply print the output to the console as a regular stream, the most visible results would be the ones at the end of the output stream, since these would be located right above the next command prompt in the terminal. Pagination, as proposed in this document, ensures that the most relevant results are shown to the users first.

	\| PackageTypes \| The package type to use to filter packages. If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. \| -PackageType `<Package Type>`\|
	\| PackageTypes \| The package type to use to filter packages (if supported by the source). If the provided package type is not a valid package type as defined by the [Package Type document](https://github.com/NuGet/Home/wiki/Package-Type-%5BPacking%5D), an empty result will returned. \| -PackageType `<Package Type>`\|


		## Who are the customers

		This targets all NuGet.exe and .NET users that want to be able to search for packages from the command line, rather than using NuGet Gallery or VS PMUI. Also, while `nuget.exe list` allows users to search different package sources for packages with a search query, the results are returned sorted by the package Id, rather than by relevance, as proposed for this Search command. Further, the Search command would provide more information on the packages than List currently does.

Created SearchCommandSpec.md #9718

Created SearchCommandSpec.md #9718

Conversation

advay26-zz commented Jun 25, 2020 • edited

dnfgituser commented Jun 25, 2020 • edited

zkat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loic-sharma Jun 25, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

donnie-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkolev92 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

advay26-zz commented Jun 25, 2020 •

edited

dnfgituser commented Jun 25, 2020 •

edited

loic-sharma Jun 25, 2020 •

edited