This repository includes some of the implementation of the paper "Efficient Meta Subspace Optimization".
Subspace optimization methods have the attractive property of reducing large-scale optimization problems to a sequence of low-dimensional subspace optimization problems. However, existing subspace optimization frameworks adopt a fixed update policy of the subspace, and therefore, appear to be sub-optimal. In this paper we propose a new \emph{Meta Subspace Optimization} (MSO) framework for large-scale optimization problems, which allows to determine the subspace matrix at each optimization iteration. In order to remain invariant to the optimization problem's dimension, we design an \emph{efficient} meta optimizer based on very low-dimensional subspace optimization coefficients, inducing a rule-based method that can significantly improve performance. Finally, we design and analyze a reinforcement learning (RL) procedure based on the subspace optimization dynamics whose learnt policies outperform existing subspace optimization methods.
- conda create -n venv_mso python=3.7
- conda activate venv_mso
- conda install pytorch cudatoolkit=10.2 -c pytorch
To reproduce the performance of the main model of our paper on the Robust Linear Regression objective as described in "Learning to Optimize", by Li and Malik (ICLR 2017):
- Run
python main.py
The model should give a 42% improvement in convergence over the baseline.
The Model repository contains the pretrained agent. Make sure to configure the script with the agent's pre-trained weights.
This repository is released under the CC-BY-NC-SA 4.0. license as found in the LICENSE file.
Please cite this work in your publications if it helps your research:
@article{choukroun2021meta,
title={Meta Subspace Optimization},
author={Choukroun, Yoni and Katz, Michael},
journal={arXiv preprint arXiv:2110.14920},
year={2021}
}