Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PPO Complex Obs/Action Space #353

Open
2 tasks done
ttumiel opened this issue Jan 18, 2023 · 3 comments · May be fixed by #359
Open
2 tasks done

PPO Complex Obs/Action Space #353

ttumiel opened this issue Jan 18, 2023 · 3 comments · May be fixed by #359

Comments

@ttumiel
Copy link

ttumiel commented Jan 18, 2023

Problem Description

Would it be useful to add a complex (nested/dictionary) action and obs space variant of the PPO algo? I did this for minerl and wondered if it would be useful to contribute into the main library? I'd happily make a PR.

Checklist

Current Behavior

Currently PPO only supports continuous or discrete actions separately and a single array observation.

Expected Behavior

PPO can support arbitrary complex action and observation spaces.

Possible Solution

  • Use tree to map over actions and observation.
  • Store arrays in the same struct shape as the obs space or flatten them for storage and unflatten when passing to the network.
@vwxyzjn
Copy link
Owner

vwxyzjn commented Jan 18, 2023

Yes that would be great. I suggest implementing it based on #338. #338 uses EnvPool's async API, which is equivalent to the regular vec env when async_batch_size = num_envs.

I was thinking about this issue more and think that you should have two types of observations:

  1. Vector obs
  2. Image obs

And for these two obs types we need to pair it with corresponding networks.

Feel free to make a PR :) Thanks.

Cc @edbeeching, this PR could help deal with Godot rl environments.

@vwxyzjn
Copy link
Owner

vwxyzjn commented Jan 23, 2023

Hi @ttumiel just following up with this. Are you still interested in the issue?

@ttumiel
Copy link
Author

ttumiel commented Jan 23, 2023 via email

@ttumiel ttumiel linked a pull request Feb 15, 2023 that will close this issue
Open
20 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add complex observation atari ppo ttumiel/cleanrl
2 participants
@vwxyzjn @ttumiel