RL4Rec

This repository contains the code used for the experiments in "State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study".

Required packages

You can install conda and then create Python 3.6 Conda environment with the required package:

$ conda env create -f ../environment.yml

Activate the environment by running conda activate rl4rec. Recommend to run it in a Linux environment.

Reproducing Experiments

We compare seven state encoders in a DQN-based RL4Rec framework when evaluating in the debiased simulations of Yahoo! R3 dataset and the coat shopping dataset.

Reproducing the results of policies with the BOI, PLD, Avg, Attention, MLP, GRU, and CNN state encoders in this paper can be done with the following commands with the best hyperparameters given for each state encoder:

On the debiased simulation of Yahoo! R3 dataset,

$ cd examples
$ python run_dqn.py --dataset yahoo --state_encoder BOI --action_dim 64 
$ python run_dqn.py --dataset yahoo --state_encoder PLD --action_dim 64 
$ python run_dqn.py --dataset yahoo --state_encoder MLP --action_dim 64 
$ python run_dqn.py --dataset yahoo --state_encoder Att --action_dim 64 --rnn_state_dim 16
$ python run_dqn.py --dataset yahoo --state_encoder MLP --action_dim 64 --activation relu
$ python run_dqn.py --dataset yahoo --state_encoder GRU --action_dim 32 --rnn_state_dim 32
$ python run_dqn.py --dataset yahoo --state_encoder CNN --action_dim 64 --rnn_state_dim 32

On the debiased simulation of the coat shopping dataset,

$ cd examples
$ python run_dqn.py --dataset coat --state_encoder BOI --action_dim 64 
$ python run_dqn.py --dataset coat --state_encoder PLD --action_dim 64 
$ python run_dqn.py --dataset coat --state_encoder MLP --action_dim 64 
$ python run_dqn.py --dataset coat --state_encoder Att --action_dim 64 --rnn_state_dim 32
$ python run_dqn.py --dataset coat --state_encoder MLP --action_dim 64 --activation tanh
$ python run_dqn.py --dataset coat --state_encoder GRU --action_dim 32 --rnn_state_dim 16
$ python run_dqn.py --dataset coat --state_encoder CNN --action_dim 64 --rnn_state_dim 64

The learning curves and evaluation results will be stored in the folder ./examples/learning_curves/ and ./examples/result/, respectively. Moreover, results reported in the paper are the averages of 10 independent runs and can be reproduced with random seed 2012 ~ 2021 with seed setting e.g., --seed 2012.

Due to the large size of Yahoo! R3 dataset, we do not upload it in this project. For using the simulation of Yahoo! R3 dataset, please download here, unzip and put it in the folder ./data/.

All the learning curves and results used in the paper are stored (first unzip ./examples/learning_curves.zip and ./examples/result.zip); and the plots used in the paper can be generated by using the following commands:

$ cd examples
$ python plots_generation.py yahoo
$ python plots_generation.py coat