  Despite their simple intuition, convolutions are more tedious to analyze than
dense layers, which complicates the generalization of theoretical and
algorithmic ideas. We provide a new perspective onto convolutions through
tensor networks (TNs) which allow reasoning about the underlying tensor
multiplications by drawing diagrams, and manipulating them to perform function
transformations, sub-tensor access, and fusion. We demonstrate this expressive
power by deriving the diagrams of various autodiff operations and popular
approximations of second-order information with full hyper-parameter support,
batching, channel groups, and generalization to arbitrary convolution
dimensions. Further, we provide convolution-specific transformations based on
the connectivity pattern which allow to re-wire and simplify diagrams before
evaluation. Finally, we probe computational performance, relying on established
machinery for efficient TN contraction. Our TN implementation speeds up a
recently-proposed KFAC variant up to 4.5x and enables new hardware-efficient
tensor dropout for approximate backpropagation.
