Ecosystem-wide test suite for solver conformance

[jaimergp]

Checklist

• I added a descriptive title
• I searched open issues and couldn't find a duplicate

What?

We propose creating a backend agnostic collection of test cases so different members of the ecosystem (e.g. conda’s own classic and libmamba solvers, but also external tools like micromamba and pixi) can assess their compliance to the underspecified solver behavior “expected” by the conda community.

Why?

• The solver tests for conda are spread across different parts of the test suite as unit and integration tests. In many cases, they incur IO overheads that are not necessary and cause a general slowdown of the test suite completion and development feedback loop.
• The PyPI ecosystem has already set a precedent in this aspect (see https://github.com/zanieb/packse and https://github.com/pradyunsg/pip-resolver-benchmarks), and the conda ecosystem should also step up.

User impact

More consistent behavior across solvers

Goals

• Set a dedicated repository with solver test cases
• Integrate it in conda/conda
• Integrate it in conda/conda-libmamba-solver

Tasks

• Create repository
• Design test case schema. Should be declarative and language agnostic (e.g. YAML or JSON)
• Create / consolidate test data (repodata, channels, etc) so there's no need to access live channels.
• Audit conda/conda for potential test cases
• Audit conda/conda-libmamba-solver for potential test cases
• Audit mamba-org/mamba for potential test cases
• Audit mamba-org/rattler for potential test cases
• Consolidate audits and write table of test cases with assumed inputs and expected outcomes
• Implement test cases

This epic is blocked by:

N/A

This epic blocks:

N/A