Use pip-resolver-benchmarks and Pip code to create "real world" resolvelib tests?

[notatallshaw]

I have an idea to use pip-resolver-benchmarks and Pip code to create tests that could exactly model real world scenarios to test resolvelib.

In particular I would want to add test cases from #134 where they are represented by a json, then as part of the test prep get the pip-resolver-benchmarks code to build the wheels, and then have enough Pip code for this to fail in the current resolvelib.

While I would try and keep it minimal as possible it probably would be quite a lot of code from both pip-resolver-benchmarks and Pip to get this working. So I understand if it's not acceptable.

In particular @pradyunsg do you have any thoughts or objections?

[uranusjr]

Sounds like a good idea to me. The suite already contains some real-world cases I stole from other resolvers before pip started using resolvelib, and now pip has its own real-world cases, it makes sense to add them here.

[notatallshaw]

I've pivoted a little on this, and am now focusing on packse tests, once I have packse tests working for pip (pypa/pip#12580) I am going to see if I can use the pip-resolver-benchmarks collection tool to make jsons that I can transform into packse jsons that could be used for packse to make "real world" tests for either pip or resolvelib or both. But I may pivot again, depending how packse and pip-resolver-benchmarks update over time.