BregmaNet : Bregman Neural Networks

BregmaNet is a PyTorch library providing multiple Bregman Neural Networks. To date, implemented models cover Bregman variants of multi-layer perceptrons and various residual networks.

Contributor: Jordan Frécon (INSA Rouen Normandy, France)

Table of Contents

1. Requirements and Installation
2. Getting Started
3. List of Supported Models
4. Citation
5. Contribution and Acknowledgments

Requirements and Installation

📋 Requirements

• PyTorch version >=1.7.1
• Python version >=3.6
• Torchvision version >=0.8.2

🔨 Installation

pip install bregmanet

In development versions can be found here.

Getting Started

⚠️ Precautions

• All images should be scaled within the domain range of the activation function.
• MLP models provided work only for 1-dimensional data inputs.
• MLP models are designed without a softmax final layer.
• All models need to be trained first. If you wish to provide your pretrained models, please contribute.

🚀 Demos

Multiple demo files can be found there in the demos folder. It contains:

• demo_toy_mlp.py: training of MLP on the Two-spiral toy dataset.
• demo_mnist_mlp.py: training of MLP on the MNIST dataset.
• demo_cifar10_resnet.py: training of ResNet20 on the CIFAR-10 dataset.
• demo_cifar100_resnet.py: training of ResNet20 on the CIFAR-100 dataset.
• demo_imagenet_resnet.py: training of ResNet18 on the ImageNet dataset.

📃 Loading a Model

To date, all Bregman neural models provided are not trained. If needed, a training procedure is made available there in the demos/utils folder. In order to load a model, proceed as follows.

Multi-Layer Perceptrons

For a sigmoid-based MLP with

• a linear input accepting 1d tensors of size 1024
• 3 hidden layers of size (1024, 1024, 512)
• a linear output layer mapping to 1d tensors of size 10

import bregmanet
model = bregmanet.MLP(activation='sigmoid', input_dim=1024, hidden_dim=[1024, 1024, 512], output_dim=10)

ResNet

For a BregmanResNet20 with SoftPlus activation function:

import bregmanet
model = bregmanet.bresnet20(activation='softplus')

List of Supported Models

The following list reports all models currently supporting a Bregman variant. If you have any issue with one of them or wish to provide your own, please contact us.

• MLP
• ResNet18
• ResNet20
• ResNet32
• ResNet34
• ResNet44
• ResNet56
• ResNet101
• Resnet110
• ResNet152
• Resnet1202
• ResNeXt50_32x4d
• ResNeXt101_32x8d
• WideResNet50_2
• WideResnet101_2

Citation

If you use this package, please cite the following work:

@inproceedings{2022_Frecon_J_p-icml_bregmanet,
  title = {{Bregman Neural Networks}},
  author = {Frecon, Jordan and Gasso, Gilles and Pontil, Massimiliano and Salzo, Saverio},
  url = {https://hal.archives-ouvertes.fr/hal-03132512},
  series    = {Proceedings of Machine Learning Research},
  booktitle = {Proceedings of the 39th International Conference on Machine Learning,
               {ICML} 2022, 17-23 July 2022, Baltimore, USA},
  year = {2022},
}

Contribution and Acknowledgments

Jordan Frecon would like to express his gratitude to the Department of Computational Statistics and Machine Learning (IIT, Genova, Italy) where part of this work was conducted during his postdoctoral position. The authors gratefully acknowledge the financial support of the French Agence Nationale de la Recherche (ANR), under grant ANR-20-CHIA-0021-01 (project RAIMO).

The proposed BregmanResNets for CIFAR-10 are based on a rework of the ResNet implementation of Yerlan Idelbayev. Other ResNet models are devised by hinging upon the official PyTorch/TorchVision repository. For more information, please refer to:

• ResNet: "Deep Residual Learning for Image Recognition"
• ResNeXt: "Aggregated Residual Transformation for Deep Neural Networks"
• WideResNet: "Wide Residual Networks"

All kind of contributions are welcome, do not hesitate to contact us!