Not-so-deep reinforcement learning

A practice module of some RL algorithms, implemented in Tensorflow Keras and OpenAI gym framework.

General:

• I tend to focus on the RL part rather than the network structure; the network structure of the algorithms in this report is pretty simple.
• I hope the generic algorithm structure helps.

Algorithms:

• A2C
• A3C
• Deep Q Network
• Categorical DQN (C51)
• Quantile DQN
• Expectile DQN

Prerequisites (testing environment)

• Python 3.8
• Numpy
• Scipy
• Tensorflow
• OpenAI-gym

examples.py contains examples for all implemented algorithms.

References

• Reinforcement Learning
• Deep Q-Learning (DQN)
• Advantage Actor Critic (A2C)
• (C51) A Distributional Perspective on Reinforcement Learning
• (ER-DQN) Statistics and Samples in Distributional Reinforcement Learning
• (QR-DQN) Distributional Reinforcement Learning with Quantile Regression
• An Analysis of Categorical Distributional Reinforcement Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Fully Parameterized Quantile Function for Distributional Reinforcement Learning
• Non-crossing quantile regression for deep reinforcement learning
• Distributional Reinforcement Learning with Maximum Mean Discrepancy

Acknowledgements:

• (PyTorch) Deep RL by Shangtong Zhang
• (Keras) Deep-RL-Keras by Hugo Germain