Abstract
Advances in Neural Information Processing Systems 34 (2020) We study the regularisation induced in neural networks by Gaussian noise
injections (GNIs). Though such injections have been extensively studied when
applied to data, there have been few studies on understanding the regularising
effect they induce when applied to network activations. Here we derive the
explicit regulariser of GNIs, obtained by marginalising out the injected noise,
and show that it penalises functions with high-frequency components in the
Fourier domain; particularly in layers closer to a neural network's output. We
show analytically and empirically that such regularisation produces calibrated
classifiers with large classification margins.