Q-Learning-LFA

This repository contains the source code to reproduce all the numerical experiments as described in the paper "Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning".

Here's a BibTeX entry that you can use to cite it in a publication:

@article{chen2019finite,
  title={Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning},
  author={Chen, Zaiwei and Zhang, Sheng and Doan, Thinh T and Clarke, John-Paul and Theja Maguluri, Siva},
  journal={arXiv e-prints},
  pages={arXiv--1905},
  year={2019}
}

Requirements

• Python (>= 3.7)
• Numpy (>= 1.19.1)

Usage

Constant Step Size

1. Show convergence of Q-learning with linear function approximation for .

cd constant_step_size
python convergence.py

2. Show exponentially fast convergence of Q-learning with linear function approximation for .

cd constant_step_size
python rate_of_convergence.py

Diminishing Step Sizes

Show convergence rate of Q-learning with linear function approximation for using diminishing step sizes .

cd diminishing_step_size
python rate_of_convergence.py

Maintainer

• Sheng Zhang - shengzhang@gatech.edu