The KFAC code was originally forked from https://github.com/gpauloski/kfac_pytorch
The VIT model was referenced from https://github.com/jeonsworld/ViT-pytorch
DeepFormer leverages several excellent libraries(hiddenlayer, torchstat, torchsummary...) to visualize the secret of deep transformers.
This code is validated to run with PyTorch v1.3
$ git clone https://github.com/closest-git/DeepFormer.git
$ cd
$ pip install .
@article{chen2021iterative,
title={An iterative K-FAC algorithm for Deep Learning},
author={Chen, Yingshi},
journal={arXiv preprint arXiv:2101.00218},
year={2021}
}