Project 2: Continuous Control

Introduction

For this project, we work with the Reacher environment.

In this environment, a double-jointed arm can move to target locations. A reward of +0.1 is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.

The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm. Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector should be a number between -1 and 1.

Solving the environment

The task is episodic, and in order to solve the environment, your agent must get an average score of +30 over 100 consecutive episodes.

Getting Started

1. Download the environment from one of the links below. You need only select the environment that matches your operating system:

   • Version 1: One (1) Agent
     • Linux: click here
     • Mac OSX: click here
     • Windows (32-bit): click here
     • Windows (64-bit): click here

   (For AWS) If you'd like to train the agent on AWS (and have not enabled a virtual screen), then please use this link (version 1) or this link (version 2) to obtain the "headless" version of the environment. You will not be able to watch the agent without enabling a virtual screen, but you will be able to train the agent. (To watch the agent, you should follow the instructions to enable a virtual screen, and then download the environment for the Linux operating system above.)

2. Place the file in the project GitHub repository, in the RL_Continous_Control/environment/ folder, and unzip (or decompress) the file.

Instructions

1. Run the train.py file to train the agent. The new weight files will be saved to RL_Continous_Control/model_weights
2. You can also use the pre-trained weights in RL_Continous_Control/model_weights and test the agent by running test_agent.py

The model.py contains the Neural Network model for the Actor and Critic. The ddpg_agent.py file contains our DDPG Agent class with different methods such as soft_update(...), act(...), learn(...) etc. along with ReplayBuffer class.

Results

The environment gets solved in 287 episodes, achieving an average score of 13.03

Dependencies

Use the requirements.txt to install the required dependencies.

pip install -r requirements.txt

References

1. DDPG paper: https://arxiv.org/pdf/1509.02971.pdf