Skip to content

Feature parity with Pytorch's BatchNormNd, code clean-up

Latest
Compare
Choose a tag to compare
@ducksoup ducksoup released this 03 Sep 14:23

This release updates ABN, InPlaceABN and InPlaceABNSync to feature parity with recent versions of Pytorch's BatchNormNd layers:

  • Add a track_running_stats parameter to enable / disable computation of running statistics independently from the layer's training state
  • Add a num_batches_tracked buffer, and allow passing momentum=None to support cumulative moving average for tracking running stats instead of exponential moving average
  • As a side-effect, now support loading parameters from standard BatchNorm without work-arounds. Still, if the loaded parameters contain negative weight elements the output will differ compared to standard BatchNorm

Additional changes:

  • Fix backward pass in eval mode: it was not properly accounting for the activation function
  • Refactor code to follow more sensible formatting standards
  • Add type annotations
  • Improve docstrings
  • Update installation instructions, pointing to the PyPI package