Doubt in the PPO.py

[Manaro-Alpha]

Shouldn't we terminate the Rollout collection after the episode is terminated?

[vwxyzjn]

Shouldn't we terminate the Rollout collection after the episode is terminated?

You don't have to wait until the episode is terminated. (What if your use case has an episode that has 400k steps?)

Please read the docs https://docs.cleanrl.dev/rl-algorithms/ppo/#overview. This is related to PPO's vectorized architecture.

[Manaro-Alpha]

No, what I meant to ask was if it terminated before minimum timesteps?

[vwxyzjn]

Please read the docs https://docs.cleanrl.dev/rl-algorithms/ppo/#overview. This is related to PPO's vectorized architecture. You want to have a fixed sized buffer to learn from.

[Manaro-Alpha]

I agree, but gym documentation states that calling env.step() after an episode has terminated returns undefined value. Won't the training throw an errror because of this undefined value or I am missing something?