PPO attention net (GTrXLNet) #176
Draft
+2,046
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
araffin
reviewed
Apr 12, 2023
Convert numpy array to torch array (for evaluation)
Remove model call when episode starts (memory dimension and features sequence not always the same)
Add assertion sanity check on batch_size and n_steps (as in PPO)
|
Hey @RemiG3 I hope everything is going well. 👋 I've been following the development of the attention PPO feature, and I'm really excited about the progress being made! Could you provide an update on the current status of this feature? I'd love to know where it stands and if there's anything new to be excited about since the last time you commented. I came across this example you shared: from sb3_contrib.ppo_attention.ppo_attention import AttentionPPO
from sb3_contrib.ppo_attention.policies import MlpAttnPolicy
VE = DummyVecEnv([lambda: gym.make("CartPole-v1")])
model = AttentionPPO(
"MlpAttnPolicy",
VE,
n_steps=240,
learning_rate=0.0003,
verbose=1,
batch_size=12,
ent_coef=0.03,
vf_coef=0.5,
seed=1,
n_epochs=10,
max_grad_norm=1,
gae_lambda=0.95,
gamma=0.99,
device='cpu',
policy_kwargs=dict(
net_arch=dict(pi=[64, 32], vf=[64, 32]),
)
)Does it still work like this? If there's any example available to better understand how this feature is being implemented or if it's already possible to test a prototype, I'd be incredibly grateful for any information in this regard. Thank you so much for the hard work you're putting into this. Many thanks, and I'm eagerly looking forward to your response. 🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add PPO attention network (GTrXLNet, paper: Stabilizing Transformers for Reinforcement Learning).
Comparisons have to be made (with the implementation of RLlib for example).
closes #165
Note: I have cleaned up most of the code, but it's still under development.
Context
Types of changes
Checklist:
make format(required)make check-codestyleandmake lint(required)make pytestandmake typeboth pass. (required)Note: we are using a maximum length of 127 characters per line.