Random sampling of n data rows

[alexclaydon]

The -n / --first-n flag was recently introduced to enable running fewer than all test cases on a given run. This runs the first n test cases. Would it be worth considering another flag that enables random sampling of n test cases from the complete set? This would be particularly convenient when testing against very large datasets stored in, e.g., .csv files. At the moment, it's necessary to copy-paste just the cases you want to run into a separate file just for that purpose.

If this feature is a good fit for promptfoo, would it be possible to make it available not just from the CLI, but also in the .yaml test config? Relatedly, would it also make sense to enable use of --first-n from the .taml config as well?

Thanks!

[typpo]

This is an interesting idea, and wouldn't be too difficult to implement 👀

Tangentially related heads up - --first-n is being renamed to --filter-first-n in the next release to match other new filtering options.