orthogonal-convolutions
This repository contains implementations and source code to reproduce experiments for the ICLR 2021 spotlight paper Orthogonalizing Convolutional Layers with the Cayley Transform by Asher Trockman and Zico Kolter.
Check out our tutorial on FFT-based convolutions and how to orthogonalize them in this Jupyter notebook.
Getting Started
You can clone this repo using:
git clone https://github.com/locuslab/orthogonal-convolutions --recursive
where the --recursive is necessary for the submodules.
The most important dependency is PyTorch >= 1.8. If you like, you can set up a new conda environment:
conda create --name orthoconv python=3.6
conda activate orthoconv
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
conda install --file requirements.txt
The Orthogonal Convolutional Layer
Our orthogonal convolutional layer can be found in layers.py. The actual layer is the module CayleyConv. It depends on the function cayley, our implementation of the Cayley transform for (semi-)orthogonalization. Additionally, CayleyConv is a subclass of StridedConv, which emulates striding functionality by reshaping the input tensor.
Running Experiments
The script train.py can be used to run most of the experiments from our paper. To try the "flagship" experiment demonstrating better clean accuracy and -norm-bounded deterministic certifiable robustness, run:
python train.py --epochs=200 --conv=CayleyConv --linear=CayleyLinear
To compare with BCOP as in our paper, run:
python train.py --epochs=200 --conv=BCOP --linear=BjorckLinear
