- Updated version of S4 module, including new measures and theory from [How to Train Your HiPPO] (state-spaces/s4#21, state-spaces/s4#54)
- Complete version of S4D module from [On the Parameterization and Initialization of Diagonal State Space Models]
- State forwarding (state-spaces/s4#49, state-spaces/s4#56)
- Support for S4 variants including DSS and GSS (documentation)
- PyTorch 1.11 had a Dropout bug which is now avoided with a custom Dropout implementation (state-spaces/s4#42, state-spaces/s4#22)
- Conjugated tensors API change in PyTorch 1.10 (state-spaces/s4#35)
- Release of Sashimi+DiffWave model (state-spaces/s4#46). Can be found at albertfgu/diffwave-sashimi
- Improved generation script for any models trained using this repository (state-spaces/s4#38)
- Re-trained SaShiMi models with the latest version of S4 (state-spaces/s4#37, state-spaces/s4#32)
- New WikiText-103 checkpoint with generation functionality (state-spaces/s4#5, state-spaces/s4#19)
- Release of new notebook (and equivalent .py file) visualizing HiPPO function reconstruction. Includes animations used in HTTYH, the Annotated S4D, and various S4 talks.
- Improved configs for Long Range Arena reported in HTTYH and S4D papers
- New datasets and ablation experiments from the S4D paper
Note that there have been various refactors and miscellaneous changes which may affect results slightly, but results should be close and general trends should hold. Feel free to file an issue for any results which do not match the papers.
- Reorganized the README and added much more documentation for using this codebase
- Minor updates to S4 modules
- By default, S4 no longer requires installing Pykeops or a custom CUDA kernel.
- New S4D (S4-diagonal) standalone model found at
src/models/sequence/ss/standalone/s4d.py. Simple variant using diagonal SSMs that recovers S4's performance on most tasks. Can be run with any existing experiment config with the additional flagmodel/layer=s4don the command line. - New LRA configs for updated S4 code, with an average score of ~86
Code release for SaShiMi audio model
Added configs for time series datasets from the Informer paper (state-spaces/s4#4)
First release of this repository containing the S4 module and configs to reproduce sCIFAR, Speech Commands, Long Range Arena, and WikiText-103 results