Liberate the requirements.txt

[ThijsvandenBerg]

Problem Description

the requirements.txt are too strict, in particular the stable-baselines3 = "1.2.0" version pinning. This makes it impossible to use cleanrl in combinations with other python modules that require more recent versions of those dependencies.

There is also version pinning in the optional requirements like:

• ale-py = {version = "0.7.4", optional = true}
• PettingZoo = {version = "1.18.1", optional = true}
• SuperSuit = {version = "3.4.0", optional = true}
• multi-agent-ale-py = {version = "0.1.11", optional = true}

Checklist

• I have installed dependencies via poetry install (see CleanRL's installation guideline.
• I have checked that there is no similar issue in the repo.
• I have checked the documentation site and found not relevant information in GitHub issues.

Current Behavior

When building our research environment we get:

ERROR: Cannot install -r requirements.txt (line 50) and stable-baselines3>=1.8.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested stable-baselines3>=1.8.0
cleanrl 1.1.0 depends on stable-baselines3==1.2.0

Expected Behavior

Non conflicting version solutions when other module have requirement that don't exactly match the pinned requirements.

Possible Solution

In particular, change the stable-baselines3 requirement to stable-baselines3>=1.2.0
In general, cleanrl should switch all dependencies to ">=" dependencies in pyproject.toml or requirements.txt, with liberal lower-bounds (the oldest valid version of a 3rd party dependency) to improve the interoperability and usability of cleanrl within other projects.

Steps to Reproduce

make a requirments.txt file or a pyproject.toml with:
stable-baselines3>=1.8.0
cleanrl >= 1.1.0

and then try to install things.

[vwxyzjn]

Hi @ThijsvandenBerg, thanks for reporting this issue. We are actively migrating to gymnasium and the latest version of SB3, but this might take a while.

Specifically regarding your request, the requirements.txt are locked because otherwise users could run into dependency issues such as #364. To resolve your use case, however, you can install the dependencies manually according to your requirement

cleanrl/pyproject.toml

Lines 15 to 49 in 7104666

	python = ">=3.7.1,<3.11"
	tensorboard = "^2.10.0"
	wandb = "^0.13.11"
	gym = "0.23.1"
	torch = ">=1.12.1"
	stable-baselines3 = "1.2.0"
	gymnasium = ">=0.28.1"
	moviepy = "^1.0.3"
	pygame = "2.1.0"
	huggingface-hub = "^0.11.1"
	ale-py = {version = "0.7.4", optional = true}
	AutoROM = {extras = ["accept-rom-license"], version = "^0.4.2", optional = true}
	opencv-python = {version = "^4.6.0.66", optional = true}
	procgen = {version = "^0.10.7", optional = true}
	pytest = {version = "^7.1.3", optional = true}
	mujoco = {version = "<=2.3.3", optional = true}
	imageio = {version = "^2.14.1", optional = true}
	free-mujoco-py = {version = "^2.1.6", optional = true}
	mkdocs-material = {version = "^8.4.3", optional = true}
	markdown-include = {version = "^0.7.0", optional = true}
	openrlbenchmark = {version = "^0.1.1b4", optional = true}
	jax = {version = "^0.3.17", optional = true}
	jaxlib = {version = "^0.3.15", optional = true}
	flax = {version = "^0.6.0", optional = true}
	optuna = {version = "^3.0.1", optional = true}
	optuna-dashboard = {version = "^0.7.2", optional = true}
	rich = {version = "<12.0", optional = true}
	envpool = {version = "^0.6.4", optional = true}
	PettingZoo = {version = "1.18.1", optional = true}
	SuperSuit = {version = "3.4.0", optional = true}
	multi-agent-ale-py = {version = "0.1.11", optional = true}
	boto3 = {version = "^1.24.70", optional = true}
	awscli = {version = "^1.25.71", optional = true}
	shimmy = {version = ">=1.0.0", extras = ["dm-control"], optional = true}

. However, things may or may not break because dependencies are not locked.

[ThijsvandenBerg]

Thanks @vwxyzjn!

After the upgrade, would you then no long pin SB3 (to the new version you're migrating to)? Pinning makes it very difficult to modules in combinations with other modules, ..and we use quite a lot of ML modules.

Another option would be that I write a unit test (not sure if that's feasible, just an idea) on the interface between cleanrl and SB3, ..then iterate over the SB3 versions to determine what versions are supported and which are causing integration issues,.. and then report those working versions back, so that the requirements can be changed to reflect that?

[elliottower]

For what it's worth, our recent releases of PettingZoo 1.23.1 and SuperSuit 3.8.0 have made the API fully consistent with Gymnasium, and shouldn't be too hard to migrate to if you are updating the code for Gymnasium already. I would be happy to help with the PettingZoo upgrades if need be, I updated the PettingZoo CleanRL tutorial to work with the current versions of PettingZoo/SuperSuit which wasn't too much work.

I agree with @ThijsvandenBerg for sure though, pinning specific versions of dependencies leads to many headaches down the road, definitely better to loosen the requirements and say greater than or equal to whichever version you have tested, and then if it breaks down the line someone can make an issue and it can be fixed. Much better that way than setting hard requirements and making it impossible to install your library with newer versions of other libraries.

[vwxyzjn]

#424 should have migrated to gymnasiuma nd upgraded SB3's version.