A multi-agent reinforcement learning (MARL) environment, trained using Proximal Policy Optimization (PPO). Learning agents Predators (red) and Prey (blue) both expend energy moving around, and replenish it by eating. Prey eat Grass (green), and Predators eat Prey if they end up on the same grid cell. Predators die of starvation when their energy is zero, Prey die either of starvation or when being eaten by a Predator. The agents asexually reproduce when energy levels of learning agents rise above a certain treshold by eating. Learning agents, learn to execute movement actions based on their partially observations of the environment to maximize cumulative reward.

Emergent Behaviors

This algorithm is an example of how elaborate behaviors can emerge from simple rules in agent-based models. Each agent (Predator, Prey, Grass) follows simple rules based on its current state, but the interactions between agents can lead to more complex dynamics at the ecosystem level. The trained agents are displaying a classic Lotka–Volterra pattern over time. This learned outcome is not obtained with a random policy:

More emergent behavior and findings are described in the config directory.

Installation

Editor used: Visual Studio Code 1.88.1 on Linux Mint 21.3 Cinnamon

1. Clone the repository:

   git clone https://github.com/doesburg11/PredPreyGrass.git

2. Open Visual Studio Code and execute:
   • Press ctrl+shift+p
   • Type and choose: "Python: Create Environment..."
   • Choose environment: Conda
   • Choose interpreter: Python 3.11.7
   • Open a new terminal
   • Install dependencies:

   pip install -r requirements.txt

3. If encountering "ERROR: Failed building wheel for box2d-py," run:

   conda install swig

   and

   pip install box2d box2d-kengz

4. Alternatively, a workaround is to copy Box2d files from assets/box2d to the site-packages directory.
5. If facing "libGL error: failed to load driver: swrast," execute:

   conda install -c conda-forge gcc=12.1.0
   

Getting started

Visualize a random policy

In Visual Studio Code run: pettingzoo/predpreygrass/random_policy.py

Training and visualize trained model using PPO from stable baselines3

Adjust parameters accordingly in: pettingzoo/predpreygrass/config/config_pettingzoo.py In Visual Studio Code run: pettingzoo/predpreygrass/train_sb3_vector_ppo_parallel.py To evaluate and visualize after training follow instructions in: pettingzoo/predpreygrass/evaluate_from_file.py

Configuration of the PredPreyGrass environment

The benchmark configuration used in the gif-video above.

References

• Terry, J and Black, Benjamin and Grammel, Nathaniel and Jayakumar, Mario and Hari, Ananth and Sullivan, Ryan and Santos, Luis S and Dieffendahl, Clemens and Horsch, Caroline and Perez-Vicente, Rodrigo and others. Pettingzoo: Gym for multi-agent reinforcement learning. 2021-2024 The utlimate go-to for multi-agent reinforcement learning deployment.
• Paper Collection of Multi-Agent Reinforcement Learning (MARL)