Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Reset options ignored when resetting due to termination / truncation from within wrapper's step #1790

Open
5 tasks done
npit opened this issue Dec 20, 2023 · 2 comments 路 May be fixed by #1805
Open
5 tasks done
Labels
bug Something isn't working documentation Improvements or additions to documentation help wanted Help from contributors is welcomed

Comments

@npit
Copy link
Contributor

npit commented Dec 20, 2023

馃悰 Bug

Given a wrapped env, options passed with the recommended way (wrapped_env.set_options) are ignored when reset is triggered by episode termination / truncation in step_wait of the env wrapper.

A (myopic) fix could be to pass self._options[env_idx] and self._seeds[env_idx] to the linked code above, or refactor a single-env resetting function to use both in DummyVecEnv.step_wait and DummyVecEnv.reset.

Apologies if this expected behavior. If so, what is the recommended way to pass reset options that affect the above resetting scenario?

To Reproduce

from stable_baselines3.common.vec_env import DummyVecEnv
import gymnasium as gym
import numpy as np


# dummy env
class CustomEnv(gym.Env):
    def __init__(self):
        super().__init__()
        self.action_space = gym.spaces.Discrete(3)
        self.observation_space = gym.spaces.Box(low=0, high=5, shape=(5,), dtype=np.uint8)

    # step terminates the episode
    def step(self, action):
        terminated = 1
        return np.zeros(5), 0, terminated, 0, {}

    # reset prints the options, if provided
    def reset(self, seed=None, options=None):
        if options is not None:
            print(" -- Options supplied:", options)
        return np.zeros(5), {}

# make and wrap the env
env = CustomEnv()
env = DummyVecEnv([lambda: env])

# resetting by invoking the wrapper function -- options are passed
print("Resetting by invoking DummyVecEnv.reset() :")
env.set_options({'opt': 1})
env.reset()

# reset by a terminating environment step:
# the wrapper step function calls self.envs[env_idx].reset(), which ignores self._options
print("Resetting by an episode-terminating invokation to DummyVecEnv.step() :")
env.set_options({'opt': 1})
env.step([0])

print("Done.")

Relevant log output / Error message

Resetting by invoking DummyVecEnv.reset() :
 -- Options supplied: {'opt': 1}
Resetting by an episode-terminating invokation to DummyVecEnv.step() :
Done.

System Info

  • OS: Linux-6.6.7-arch1-1-x86_64-with-glibc2.34 # 1 SMP PREEMPT_DYNAMIC Thu, 14 Dec 2023 03:45:42 +0000
  • Python: 3.8.15
  • Stable-Baselines3: 2.2.1
  • PyTorch: 1.13.1+cu117
  • GPU Enabled: True
  • Numpy: 1.24.1
  • Cloudpickle: 2.2.1
  • Gymnasium: 0.28.1

Checklist

  • My issue does not relate to a custom gym environment. (Use the custom gym env template instead)
  • I have checked that there is no similar issue in the repo
  • I have read the documentation
  • I have provided a minimal and working example to reproduce the bug
  • I've used the markdown code blocks for both code and stack traces.
@npit npit added the bug Something isn't working label Dec 20, 2023
@araffin araffin added the Maintainers on vacation Maintainers are on vacation so they can recharge their batteries, we will be back soon ;) label Dec 20, 2023
@araffin araffin added help wanted Help from contributors is welcomed and removed Maintainers on vacation Maintainers are on vacation so they can recharge their batteries, we will be back soon ;) labels Jan 10, 2024
@araffin
Copy link
Member

araffin commented Jan 10, 2024

Hello,
thanks for reporting the bug =)

I would happy to receive a PR that fixes this issue.
We also need to document the behavior, because we need to decide what to do with the other options (shall we wait for each env to reset or discard them?).

@araffin araffin added the documentation Improvements or additions to documentation label Jan 10, 2024
@npit
Copy link
Contributor Author

npit commented Jan 11, 2024

I'll have a go at it. Presumably a similar fix is required for the SubprocVecEnv, as the parameters are missing there as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Improvements or additions to documentation help wanted Help from contributors is welcomed
Projects
None yet
2 participants