Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model

International Conference on Learning Representation (ICLR), 2024

[Project Page] [Arxiv] [Openreview]

Yinan Zheng*, Jianxiong Li*, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu

The official implementation of FISOR, which represents a pioneering effort in considering hard constraints (Hamilton-Jacobi Reachability) within the safe offline RL setting.

Methods

FISOR transforms the original tightly-coupled safety-constrained offline RL problem into three decoupled simple supervised objectives:

• Offline identification of the largest feasible region;
• Optimal advantage learning;
• Optimal policy extraction via time-independent classifier-guided diffusion model, enhancing both performance and stability.

Branches Overview

Branch name	Usage
master	FISOR implementation for Point Robot, Safety-Gymnasium and Bullet-Safety-Gym; data quantity experiment; feasible region visualization.
metadrive_imitation	FISOR implementation for MetaDrive; data quantity experiment; imitation learning experiment.

Installation

conda create -n env_name python=3.9
conda activate FISOR
git clone https://github.com/ZhengYinan-AIR/FISOR.git
cd FISOR
pip install -r requirements.txt

Main results

Run

# OfflineCarButton1Gymnasium-v0
export XLA_PYTHON_CLIENT_PREALLOCATE=False
python launcher/examples/train_offline.py --env_id 0 --config configs/train_config.py:fisor

where env_id serves as an index for the list of environments.

Data Quantity Experiments

We can run filter_data.py to generate offline data of varying volumes. We also can download the necessary offline datasets (Download link). Then run

python launcher/examples/train_offline.py --env_id 17 --config configs/train_config.py:fisor --ratio 0.1

where ratio refers to the proportion of the processed data to the original dataset.

Feasible Region Visualization

We need to download the necessary offline dataset for Point Robot environment (Download link). Training FISOR in the Point Robot environment

python launcher/examples/train_offline.py --env_id 29 --config configs/train_config.py:fisor

Then visualize the feasible region by running viz_map.py.

Bibtex

If you find our code and paper can help, please cite our paper as:

@article{zheng2024safe,
  title={Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model},
  author={Zheng, Yinan and Li, Jianxiong and Yu, Dongjie and Yang, Yujie and Li, Shengbo Eben and Zhan, Xianyuan and Liu, Jingjing},
  journal={arXiv preprint arXiv:2401.10700},
  year={2024}
}

Acknowledgements

Parts of this code are adapted from IDQL and DRPO.