spectral-neural-nets

Reproduction, Colab Notebooks

Your can reproduce my experimental results all from Colab:

• Cats vs Dogs Benchmark
• Filter Visualization
• Spectral Encoder/Spatial Decoder autoencoder

Overview

The center-piece of the spectral_neural_nets package is the ParametricFourierConvolutionBase. This class performs pointwise convolution in the spectral domain for a given parametric as described in the publication.

The ParametricFourierConvolutionBase layer expects inputs to be batches of complex image tensors. Unfortunately, Tensorflow does not currently support automatic differentiation for tf.einsum when used on complex numbers. Due to this, the ParametricFourierConvolutionBase expects inputs to be of the shape (None, height, width, channels, 2). The final dimension is used to store the real and imaginary portions of the tensor respectively.

For convenience, spectral_neural_nets also provides fft_layer, ifft_layer, from_complex, to_complex to easily convert between representations.

from spectral_neural_nets.layers import Gaussian2DFourierLayer, fft_layer, from_complex
inputs = Input(shape=(150, 150, 3))
# shape=(150, 150, 3), dtype=tf.float32
x = fft_layer(inputs)
# shape=(150, 76, 3), dtype=tf.complex32
x = from_complex(x)
# shape=(150, 76, 3, 2), dtype=tf.float32
x = Gaussian2DFourierLayer(filters)(x)
# shape=(150, 76, 3, 2), dtype=tf.float32
x = to_complex(x)
# shape=(150, 76, 3), dtype=tf.complex32
x = ifft_layer(x)
# shape=(150, 150, 3), dtype=tf.float32

Fully implemented Linear and Gaussian Parametric Spectral layers are included in the package.

The package also provides a fully trainable FourierDomainConv2D as described by Pratt et al.

Installation

git clone https://github.com/lukewood/spectral-neural-nets \
    && cd spectral-neural-nets \
    && python setup.py develop

Citation

If you use this repo, cite the paper:

@inproceedings{wood2021parametric,
  title={Parametric Spectral Filters for Fast Converging, Scalable Convolutional Neural Networks},
  author={Wood, Luke and Larson, Eric C},
  booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={2800--2804},
  year={2021},
  organization={IEEE}
}