Planning

[bstee615]

Components

• Main agent
• Adversarial actions
• Adversarial training (see Section 3.2 and Formula 4)

Tasks

• Implement the main agent with Trust Region Policy Optimization (TRPO, see Link)
  • Use InvertedPendulum environment in OpenAI Gym
  • Use Stable Baselines implementation for TRPO
  • Use hyperparameters below
• Extend InvertedPendulum's Gym environment with adversarial actions
  • See new environment docs, MuJoCoPy docs, and source
• Add adversarial training to the main agent training loop (see Section 3.3, Algorithm 1)

Original paper implementation Link

Libraries

• OpenAI Gym & MuJoCo
  • Tasks are InvertedPendulum, HalfCheetah, Swimmer, Hopper, Walker2d, and Ant
• Custom Gym environment for adversarial actions
• Custom training code built with rllab

Training setting (see Section 4.1)

• Policy network
  • 2 hidden layers
  • 64 neurons each layer
• Training
  • 100 iterations on InvertedPendulum
  • 500 iterations for other tasks
• Hyperparameters of TRPO are selected by grid search