[???? Coming soon! Official release with improved docs. Stay tuned. ????]
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
ViViT is a collection of numerical tricks to efficiently access curvature from
the generalized Gauss-Newton (GGN) matrix based on its low-rank structure.
Provided functionality includes computing
- GGN eigenvalues
- GGN eigenpairs (eigenvalues + eigenvector)
- 1ˢᵗ- and 2ⁿᵈ-order directional derivatives along GGN eigenvectors
- Newton steps
These operations can also further approximate the GGN to reduce cost via
sub-sampling, Monte-Carlo approximation, and block-diagonal approximation.
How does it work? ViViT uses and extends
BackPACK for
PyTorch. The described functionality is
realized through a combination of existing and new BackPACK extensions and
hooks into its backpropagation.
Installation
???? The PyPI release is coming soon. ????
For now, you need to install from GitHub via
pip install vivit-for-pytorch@git+https://github.com/f-dangel/vivit.git#egg=vivit-for-pytorch
Examples
???? Coming soon! ????
How to cite
If you are using ViViT, consider citing the paper
@misc{dangel2022vivit,
title={{ViViT}: Curvature access through the generalized Gauss-Newton's low-rank structure},
author={Felix Dangel and Lukas Tatzel and Philipp Hennig},
year={2022},
eprint={2106.02624},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
f-dangel
