Hyperparameter Optimization Module #151

middleyuan · 2024-04-10T13:22:06Z

The pull request contains

DDPG, SAC, PPO

Change the saving method for the checkpoint.
Add activation as an argument for the constructor.

GPMPC

Add the feature that allows one to learn GP in parallel (this speeds up roughly D times faster, where D is the dimension of the dynamics)
Add activation as an argument for the constructor ('Matern' or 'RBF' kernel are supported).

Hyperparameter optimization (HPO) module

database.py implements the functions that interact with MySQL.
hpo_sampler.py defines the hyperparameter spaces for the selected controller.
hpo.py defines the hyperparameter optimization class.

safe_control_gym/hyperparameters/README.md describes the basic steps for running HPO.

…ve unnecessary config in cartpole_stab.yaml 2. add hpo module in test_build.py

… of JSRL on PPO.

…ckpoint in ppo.py. 4. Boolean var in ppo_sampler.

adamhall · 2024-04-10T20:38:26Z

Run precommit hooks etc. to ensure formatting is good.

adamhall · 2024-04-10T20:40:34Z

experiments folder was removed. Clean up experiments and move to examples folder that exists in the current main branch.

adamhall · 2024-04-10T20:41:23Z

Move any files used for tests into the tests directory.

adamhall · 2024-04-10T20:42:19Z

rename hyperparameters_13.....yaml to something more readable.

…ts to tests. 3. write comments in batch GP in GPMPC controller. 4. move experiments in examples.

… and pass this version of pre-commit hooks.

…dd an example of hpo for gpmpc.

… avg return in base_experiment.py. 3. use BaseExperiment class in hpo example 3. add hp study bash script and jupyter notebook for gpmpc.

… config. 3. add done_on_max_steps in base_experiment.py. 4. remove _run() and use BaseExperiment in hpo.

Federico-PizarroBejarano

I pulled your branch and reran the pre-commit hooks and there are some minor fixes you should add.

.gitignore

.pre-commit-config.yaml

examples/hpo/gp_mpc/config_overrides/cartpole/gp_mpc_cartpole_150.yaml

examples/hpo/rl/ppo/config_overrides/cartpole/ppo_cartpole_.yaml

examples/no_controller/verbose_api.py

safe_control_gym/hyperparameters/README.md

safe_control_gym/hyperparameters/hpo.py

Federico-PizarroBejarano

LGTM

adamhall

LGTM

adamhall · 2024-04-10T20:26:23Z

safe_control_gym/controllers/mpc/gp_mpc.py

@@ -815,6 +832,128 @@ def learn(self,
        except UnboundLocalError:
            training_results['info'] = None
        return training_results
+
+    def _learn(self):


Change def learn to something like def gp_hyperparam_opt and then change def learn and def _learn to more closely match _learn from the other learning classes
.

adamhall · 2024-04-10T20:29:22Z

safe_control_gym/controllers/mpc/gp_mpc.py

+
+        return train_runs, test_runs
+
+    def _run(self):


Ideally, we standardize this across all the controllers, if possible, and use the experiment class instead of _run.

adamhall · 2024-04-10T20:35:26Z

pyproject.toml

+optuna-dashboard = "^0.9"
+mysql-connector-python = "8.0.33"
+pymysql = "1.0.3"
+seaborn = "0.12.2"


Do we need this? Delete if matplotlib is sufficient

adamhall · 2024-04-10T20:48:01Z

safe_control_gym/controllers/mpc/gp_utils.py

@@ -395,6 +508,293 @@ def kernel_inv(self,
        non_lazy_tensors = [k.inv_matmul(torch.eye(num_of_points).double()) for k in k_list]
        return torch.stack(non_lazy_tensors)

+class GaussianProcesses:
+    '''Gaussian Processes decorator for batch GP in gpytorch.


Rename or Add a comment that this is for parallel training of GPs and that only 1 learning rate can be specified for all GPs

Still ToDo.

middleyuan added 30 commits June 30, 2023 18:29

1. bug fixed. 2. kernel extension. 3. batch GP implementatoin.

8d3be23

update dependencies

7f0e3ff

explicitliy import scipy.linalg

cf3d4e8

add cartpole configs for gpmpc

359eecc

add hyperparameter optimization module

89a29b8

catch all the exception in hpo for debugging purpose.

9e2a7ef

put cartpole configs for gpmpc under the folder of gpmpc

27454da

add hpo scripts

17e408b

1. include pandas 2. change rel import in gpmpc_experiment.py 3. remo…

bb4e1b0

…ve unnecessary config in cartpole_stab.yaml 2. add hpo module in test_build.py

rename config to match default algo name.

14a6db1

remove old configs

e6a2e3d

add tests

0484388

edit bash file with correct arg name

84830df

add another host in gpmpc_hpo.sh

e69e048

change to new dir in gpmpc_hpo.sh

097e1c2

1. fix a small bug 2. add test_train_gpmpc_cartpole

405dcea

add a hpo parallelism test

549ff3e

saving before runing hpo

81b5602

I think the bug is that it reaches thee goal in the first step.

a5ad5f2

1. PPO configs. 2. Make cartpole init states harder. 3. First version…

ce4d75e

… of JSRL on PPO.

Re-organize a bit (file name, remove __init__.py in test folders).

b40566c

1. HPO strategies. 2. test on hpo for ppo. 3. another way to save che…

23f571d

…ckpoint in ppo.py. 4. Boolean var in ppo_sampler.

update gitignore

802edb6

change configs

02d1c33

update bash for hpo on gpmpc

20d3a7f

add prior arg in gpmpc_sampler

ad96f6f

1. HPO effort evaluations. 2. Bash file for hpo strategy evalution.

5318c25

update dependencies

924d3b3

add the freedom to choose between random sampler and TPE sampler.

14ae2aa

1. add strategy 5. 2. add unit test accordingly.

c0b1b34

middleyuan added 3 commits April 10, 2024 13:59

test training with given optimized hp files.

83ec989

1. test hpo with and without MySQL. 2. update README.

e890c04

remove discrepancy of readme.

e9499cd

middleyuan added the enhancement New feature or request label Apr 10, 2024

middleyuan requested a review from Federico-PizarroBejarano April 10, 2024 13:22

update readme

c45a975

middleyuan added 14 commits April 15, 2024 16:04

1. remove 'pandas' and 'seaborn' in package dependencies. 2. move tes…

8350b63

…ts to tests. 3. write comments in batch GP in GPMPC controller. 4. move experiments in examples.

-

ee9ec34

update config_overrides in examples of rl

a260695

run pre-commit hooks to improve linting

13ef164

1. ignore W503 and W504 as they conflict in pre-commit-config. 2. run…

3bbd0ba

… and pass this version of pre-commit hooks.

add activation config to the examples that use RL.

56f2738

1. standardize hpo template in the examples. 2. remove _learn(). 3. a…

ba837c5

…dd an example of hpo for gpmpc.

run pre-commit hooks.

51c601e

add gpmpc hpo test without using mysql

fb572f2

1. update config of cartpole task. 2. add max_steps and exponentiated…

c5fbeed

… avg return in base_experiment.py. 3. use BaseExperiment class in hpo example 3. add hp study bash script and jupyter notebook for gpmpc.

1. add bash files to automate hpo pipeline for gpmpc. 2. update gpmpc…

f8e3d0c

… config. 3. add done_on_max_steps in base_experiment.py. 4. remove _run() and use BaseExperiment in hpo.

Merge remote-tracking branch 'upstream/main' into hpo

1f62d3b

match .gitignore to upstram/main.

ffb29de

update for review

1508634

Federico-PizarroBejarano reviewed May 5, 2024

View reviewed changes

update based on the review comments.

896ac9f

Federico-PizarroBejarano approved these changes May 14, 2024

View reviewed changes

fix typo in readme.

3b49f7b

adamhall approved these changes May 14, 2024

View reviewed changes

Federico-PizarroBejarano merged commit f90ac22 into utiasDSL:main May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hyperparameter Optimization Module #151

Hyperparameter Optimization Module #151

middleyuan commented Apr 10, 2024

adamhall commented Apr 10, 2024

adamhall commented Apr 10, 2024

adamhall commented Apr 10, 2024

adamhall commented Apr 10, 2024

Federico-PizarroBejarano left a comment

Federico-PizarroBejarano left a comment

adamhall left a comment

adamhall Apr 10, 2024

adamhall Apr 10, 2024

adamhall Apr 10, 2024

adamhall Apr 10, 2024

adamhall May 7, 2024

Hyperparameter Optimization Module #151

Hyperparameter Optimization Module #151

Conversation

middleyuan commented Apr 10, 2024

adamhall commented Apr 10, 2024

adamhall commented Apr 10, 2024

adamhall commented Apr 10, 2024

adamhall commented Apr 10, 2024

Federico-PizarroBejarano left a comment

Choose a reason for hiding this comment

Federico-PizarroBejarano left a comment

Choose a reason for hiding this comment

adamhall left a comment

Choose a reason for hiding this comment

adamhall Apr 10, 2024

Choose a reason for hiding this comment

adamhall Apr 10, 2024

Choose a reason for hiding this comment

adamhall Apr 10, 2024

Choose a reason for hiding this comment

adamhall Apr 10, 2024

Choose a reason for hiding this comment

adamhall May 7, 2024

Choose a reason for hiding this comment