Skip to content

Latest commit

 

History

History
474 lines (424 loc) · 21.2 KB

changelog.md

File metadata and controls

474 lines (424 loc) · 21.2 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

1.6.0 - 2023-06-26

With this patch, BackPACK supports torch==2.x and deprecates python3.7 (PR). BackPACK now supports BCEWithLogitsLoss, and supports KFAC for N-dimensional (transpose) convolutions.

Added/New

  • Example use case for retain_graph (PR)
  • Example for computing batched Jacobians (PR)
  • Support BCEWithLogitsLoss
    • BatchDiagHessian and DiagHessian extensions (PR)
    • BatchDiagGGN (exact, MC) and DiagGGN (exact, MC) extensions (PR)
    • KFAC extension (PR)
    • KFLR extension (PR)
  • Extend KFAC, KFLR, KFRA to N-dimensional convolutions (and transpose convolutions) (blog post)
    • Support for Conv{1,2,3}d (PR)
    • Support for ConvTranspose{1,2,3}d (PR)

Fixed/Removed

  • Use correct imports on website code samples (PR)

Internal

  • Update code to latest black and flake8 (PR)
  • Fix examples on splash page (PR)
  • Abstract negative log-likelihood losses in core (PR)
    • Apply to MSELoss derivatives (PR)
    • Apply to CrossEntropyLoss derivatives (PR)
    • Implement BCEWithLogitsLoss derivatives (PR)
  • Implement second-order derivatives of BCEWithLogitsLoss (PR)
  • Implement optimized sampled gradients for BCEWithLogitsLoss (PR)
  • Use batch size 1 in KFAC tests for ResNets (PR)
  • Test with PyTorch 1.9.0 and 1.12.0 and make tests compatible (PR)
  • Use unfoldNd package
    • For input unfolding of convolutions (PR)
    • For input unfolding of transpose convolutions (PR)
  • Fully-document utility functions for convolutions (PR)
  • Make output shape of unfolded input of transpose convolution consistent with convolution case (PR)
  • Fully-document HBPConv2d class (PR)
  • Fix support for PyTorch 1.13 (PR)
  • Update linting and formatting to latest black, flake8 (PR)

1.5.2 - 2022-12-19

This patch adds support for torch>=1.13.0 to BackPACK.

1.5.1 - 2022-11-03

This patch fixes temporary compatibility issues with the latest PyTorch release.

Fixed/Removed

  • Circumvent compatibility issues with torch==1.13.0 by requiring `torch<1.13.0`` (PR)

1.5.0 - 2022-02-15

This small release improves ResNet support of some second-order extensions and adds some more custom layers.

Thanks to @jabader97 for contributing to KFAC.

Added/New

  • KFAC for ResNets modules (commit)
  • Custom modules for
  • {Batch}DiagGGN{Exact,MC}, {Batch}DiagHessian support for
    • Custom slicing module (commit)
    • Custom padding module (commit)
  • SqrtGGN{Exact,MC} support for
    • BatchNormNd (eval mode) (commit)
    • ResNet modules (ScaleModule, SumModule, Identity) (commit)
    • Custom slicing module (commit)
    • Custom padding module (commit)

Fixed/Removed

Internal

  • Automate PyPI release through GitHub action (commit)
  • Add KFAC tests for linear layers to test suite (commit)
  • Update formatting to black==22.1.0 (commit)
  • Add BackPACK logo to RTD (commit)

1.4.0 - 2021-11-12

This release ships many new features. Some rely on recent PyTorch functionality. We now require torch>=1.9.0.

Highlights:

  • ResNets & RNNs: Thanks to @schaefertim for bringing basic support for RNNs (#16, tutorial) and ResNets (#14, tutorial)
  • SqrtGGN{Exact,MC} extension: Symmetric factorization of the generalized Gauss-Newton/Fisher (see arXiv:2106.02624)
  • Sub-sampling: Allows for restricting BackPACK extensions to a sub-set of samples in the mini-batch (#12, tutorial)

Added/New

  • Converter functionality for basic support of ResNets and RNNs [PR1, PR2, PR3]
  • New extensions:
    • SqrtGGNExact: Symmetric factorization of the exact GGN/Fisher [PR]
    • SqrtGGNMC: Symmetric factorization of the MC-approximated GGN/Fisher [PR]
  • Module support:
    • Linear: Support additional (more than 2) input dimensions [PR1, PR2]
    • BatchNormNd: Distinguish evaluation and training mode, support first-order extensions and DiagGGN{Exact,MC} [#160, PR1, PR2]
    • AdaptiveAvgPoolND: Support first-order extensions and DiagGGN{Exact,MC} [PR]
    • RNN: Support first-order extensions and DiagGGN{MC,Exact} [PR1 PR2 PR3]
    • LSTM: Support first-order extensions and DiagGGN{MC,Exact} [PR]
    • CrossEntropyLoss: Support additional (more than 2) input dimensions. [PR]
    • Embedding: Support first-order extensions and DiagGGN{MC,Exact} [PR]
  • Mini-batch sub-sampling
    • BatchGrad [PR1, PR2]
    • SqrtGGN{Exact,MC} [PR]
  • retain_graph option for backpack context [PR]
  • Assume batch axis always first [PR]

Fixed/Removed

  • Deprecate python3.6, require at least python3.7 [PR]

Internal

  • Use full_backward_hook for torch>=1.9.0 [PR]
  • Core
    • Implement derivatives for LSTM [PR]
    • Implement derivatives for AdaptiveAvgPoolNd [PR]
    • Sub-sampling
      • weight_jac_t_mat_prod [PR]
      • bias_jac_t_mat_prod [PR]
      • *_jac_t_mat_prod of RNN and LSTM parameters [PR]
      • jac_t_mat_prod [PR]
      • Hessian square root decomposition (exact and MC) [PR]
    • Refactor: Share code for *_jac_t_mat_prod [PR]
  • Extensions
    • Refactor BatchL2Grad, introducing a base class [PR]
    • Automate parameter functions for BatchGrad and Grad [PR]
    • Introduce interface to check module hyperparameters [PR]
  • Tests
    • Check if module Hessian is zero [PR]
    • Reduce run time [PR]

1.3.0 - 2021-06-16

Thanks to @sbharadwajj and @schaefertim for co-authoring many PRs shipped in this release.

Added

  • New extensions
    • BatchDiagGGN{Exact,MC}: Per sample diagonal of the GGN/Fisher, exact or with a Monte-Carlo approximation [PR1, PR2, PR3, example]
    • BatchDiagHessian: Per sample diagonal of the Hessian [PR1, PR2, example]
  • Support for more layers ([PR, overview])
    • DiagGGN{Exact,MC} extensions
      • Conv{1,3}d, ConvTranspose{1,2,3}d, LeakyReLU, LogSigmoid, ELU, SELU [PR]
      • MaxPool{1,3}d [PR]
      • AvgPool{1,3}d [PR]
    • DiagHessian extension
      • Conv{1,3}d, ConvTranspose{1,2,3}d, LeakyReLU, LogSigmoid [PR]
      • MaxPool{1,3}d [PR]
      • AvgPool{1,3}d [PR]
      • ELU, SELU [PR]
    • group argument of (transpose) convolutions
      • Full support for first-order diagonal curvature extensions [PR1, PR2, PR3, PR4]
      • No support (yet) for KFAC, KFLR and KFRA extensions [PR]
  • Extension hook which allows to run code right after a BackPACK extension [PR, example]
  • Context to disable BackPACK [PR]
  • Tutorial how to extend custom modules [PR, example]
  • (Experimental) Alternative convolution weight Jacobian with option to save memory [PR, example]

Fixed/Removed

  • Remove hooks that save input/output shapes. This probably resolves #97 [PR]
  • Remove DiagGGN from API (use DiagGGNExact instead). It was indented as abstract parent class for DiagGGNExact and DiagGGNMC [PR]

Internal

  • CI
    • Move tests from Travis to GitHub actions [PR, small fix]
    • Test DiagHessian with new test suite [PR]
    • Test DiagGGN with new test suite, introduce 'light' and 'full' tests [PR1, PR2]
    • Fix isort [PR]
    • Add partial docstring checks [PR]
    • Add docstrings to contribution guide lines [commit]
    • Auto-format and lint examples [PR]
  • Refactoring
    • Share code between Conv{Transpose}{1,2,3}d in BatchL2Grad [PR]
    • Use eingroup package, remove custom eingroup utility [PR]
  • Core
    • Implement derivatives for MaxPool{1,3}d [PR]
    • Implement derivatives for AvgPool{1,3}d [PR]
    • Support for groups in (transpose) convolutions [PR1, PR2, PR3, PR4]

1.2.0 - 2020-10-26

Thanks to @sbharadwajj for co-authoring many PRs shipped in this release.

Added

  • Deprecated python3.5, tested compatibility with PyTorch 1.6.0 [PR]
  • Support first-order extensions for Conv1d, Conv3d, ConvTranspose1d, ConvTranspose2d, ConvTranspose3d
    • extensions.BatchGrad [PR]
    • extensions.BatchL2Grad [PR]
    • extensions.SumGradSquared and extensions.Variance [PR]
    • Raise exceptions for unsupported exotic hyperparameters [PR1, PR2]
  • New example: Backpropagating through BackPACK quantities [commit]
  • New extensions in API: Block-diagonal curvature products
    • Exposed via extensions.HMP, extensions.GGNMP, extensions.PCHMP [PR]
    • Examples: Hutchinson trace estimation [PR] and Hessian-free optimization with CG [PR]

Fixed

  • Add missing zero_grad in the diagonal GGN second-order optimization example [PR]

Internal

  • Increased test coverage
    • New test suite for backpack.extensions [PR]
    • New test suite for backpack.core [PR]
  • Implemented derivatives of the following operations in backpack.core
    • More activation functions [PR]
    • Conv1d, Conv3d [PR]
    • ConvTranspose1d, ConvTranspose2d, ConvTranspose3d [PR]
  • Refactor firstorder extensions to share more code [PR1, PR2]
  • Removed detachs to support differentiating through quantities [PR]

1.1.1 - 2020-04-29

Added

  • Improved documentation, moved to ReadTheDocs [PR1, PR2, PR3]
  • Tested compatibility with PyTorch 1.5.0.
  • Support 2nd-order backprop for vectors in MSELoss [PR]
  • Sanity checks to raise warnings if the following are used. inplace modification [PR], unsupported loss parameters [PR], custom losses in 2nd-order backpropagation [PR]

Fixed

  • Removed opt_einsum dependency [PR]
  • Missing implementations and wrong backpropagation of KFRA for Conv2d, MaxPool2d, and AvgPool2d [PR]
  • Remove try_view and use reshape to use PyTorch 1.4.0 improvements [PR]

Internal

  • Docstring style [PR]

1.1.0 - 2020-02-11

Added

  • Support MC sampling [Issue, PR]
  • Utilities to handle Kronecker factors [PR]
  • Examples [PR]

Fixed

  • Fixed documentation issue in Batch l2 [PR]
  • Added support for stride parameter in Conv2d [Issue, PR]
  • Pytorch 1.3.0 compatibility [PR, PR]

Internal

  • Added continuous integration [PR], test coverage [PR], style guide enforcement [PR]
  • Changed internal shape conventions of backpropagated quantities for performance improvements [PR]

1.0.1 - 2019-09-05

Fixed

  • Fixed PyPI installaton

1.0.0 - 2019-10-03

Initial release