[???? Coming soon! Official release with improved docs. Stay tuned. ????]

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

ViViT is a collection of numerical tricks to efficiently access curvature from
the generalized Gauss-Newton (GGN) matrix based on its low-rank structure.
Provided functionality includes computing

  • GGN eigenvalues
  • GGN eigenpairs (eigenvalues + eigenvector)
  • 1ˢᵗ- and 2ⁿᵈ-order directional derivatives along GGN eigenvectors
  • Newton steps

These operations can also further approximate the GGN to reduce cost via
sub-sampling, Monte-Carlo approximation, and block-diagonal approximation.

How does it work? ViViT uses and extends
BackPACK for
PyTorch. The described functionality is
realized through a combination of existing and new BackPACK extensions and
hooks into its backpropagation.

Installation

???? The PyPI release is coming soon. ????

For now, you need to install from GitHub via

pip install [email protected]+https://github.com/f-dangel/vivit.git#egg=vivit-for-pytorch

Examples

???? Coming soon! ????

How to cite

If you are using ViViT, consider citing the paper

@misc{dangel2022vivit,
      title={{ViViT}: Curvature access through the generalized Gauss-Newton's low-rank structure},
      author={Felix Dangel and Lukas Tatzel and Philipp Hennig},
      year={2022},
      eprint={2106.02624},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
GitHub - f-dangel/vivit at pythonawesome.com
Curvature access through the generalized Gauss-Newton’s low-rank structure: Eigenvalues, eigenvectors, directional derivatives & Newton steps - GitHub - f-dangel/vivit at pythonawesome.com