To watch the video, click on the screenshot below (right-click and select "Open link in new tab" to view in a new tab):
If you wish to download the video directly, click here.
This repository contains the datasets and code for the paper DCRL-Green: Sustainable Data Center Environment and Benchmark for Multi-Agent Reinforcement Learning.
DCRL-green is a framework for testing multi-agent Reinforcement Learning (MARL) algorithm that optimizes data centers for multiple objectives of carbon footprint reduction, energy consumption, and energy cost. It uses OpenAI Gym standard and supports modeling and control of three different types of problems: Carbon aware flexible load shifting, Data center HVAC cooling energy optimization and carbon aware battery auxiliary supply.
Main contributions of DCRL-Green:
- the first OpenAI framework, in our knowledge, focused on carbon footprint reduction for data centers
- modular design meaning users can utilize pre-defined modules for load shifting, energy and battery or build their own
- scalable architecture that allows multiple different types of modules and connections between them
- robust data center model that provides in-depth customization to fit users' needs
- provides pre-defined reward functions as well as interface to create custom reward functions
- built-in mechanisms for reward shaping focused on degree of cooperation between the agents and level of prioritization of carbon footprint reduction versus energy cost
- custom reward shaping through custom reward functions
- build-in MARL algorithms, with ability to incorporate user-specified custom agents
Currently, we provide two versions for the data center dynamics.
DCRL (dcrl_env.py)
: This default version is implemented in Python and can be used with the prerequisites listed below.
DCRLeplus (dcrl_eplus_env.py)
: This uses the EnergyPlus model of a data center from the Sinergym repository. We provide a docker image for this environment as well as instructions for manual installation.
Real-time Carbon Footprint Minimization in Sustainable Data Centers wth Reinforcement Learning. Best Paper - Best ML Innovation award at NeurIPS Climate Change AI Workshop.
Refer to the docs for documentation of the DCRL-Green.
- Linux OS (Ubuntu 20.04)
- Conda
- [Optional] Docker (see docs) (Only required for the Sinergym-EnergyPlus Environment)
First, download the repository. If using HTML, execute:
$ git clone https://github.com/HewlettPackard/dc-rl.git
If using SSH, execute:
$ git clone git@github.com:HewlettPackard/dc-rl.git
Make sure you have conda installed. For more instructions on installing conda please check the documentation.
Change the current working directory to the dc-rl folder:
$ cd dc-rl
Create a conda environment and install dependencies:
$ conda create -n dcrl python=3.10
$ conda activate dcrl
$ pip install -r requirements.txt
Make sure you are inside the dc-rl
directory first.
To install the DCRLeplus environment using a docker image (recommended) run the following command to pull the image:
$ docker pull agnprz/carbon_sustain:v3
To install DCRLeplus manually (not recommended), you need to follow the instructions to manually install Sinergym. Make sure all the required components (custom Python environment, EnergyPlus, and BCVTB) are correctly installed.
Before running the DCRL environment, make sure you are in the dc-rl
folder. If you are in your home directory, run cd dc-rl
or cd PATH_TO_PROJECT
depending on where you downloaded the GitHub repository.
To run an episode of the environment with a random agent execute:
$ python dcrl_env.py
To start training, run the following command:
(Note: The episode_reward_mean
will be nan
for the first few iterations until 1 episode is completed)
For PPO:
$ python train_ppo.py
For MADDPG:
$ python train_maddpg.py
For A2C:
$ python train_a2c.py
First, run the docker image that you previosuly downloaded:
$ docker run -t -i -v $PWD:/sinergym/dc-rl --shm-size=10.24gb agnprz/carbon_sustain:v3
Finally to run DCRLeplus use:
$ cd dc-rl
$ EPLUS=1 python train_ppo.py
Note that this will use PPO
agents; for MADDPG
use the train_maddpg.py
Python script and for A2C
use the train_a2c.py
script. Other algorithms can be used, it is only necessary to utilize the RLLib algorithms.
If you want to run the DCRL-Green framework in background mode use the following command:
$ nohup python PYTHON_SCRIPT > OUTPUT_FILE.txt &
where PYTHON_SCRIPT
is the script you want to run (e.g., train_ppo.py
) and OUTPUT_FILE
is the name of the file that will contain the output (e.g. latest_experiment_output.txt
).
Monitor the training using TensorBoard. By default, the location of the training data is at ./results
. To visualize, run:
$ tensorboard --logdir ./results
Contributions are welcome. For major changes, please open an issue first to discuss what you would like to change. Please ensure to update tests as appropriate.
For any questions or concerns, please open an issue in this repository.