Reexamining the principle of mean-variance preservation for neural network initialization

Reexamining the principle of mean-variance preservation for neural network initialization

Blog Article

Before backpropagation training, it is common to randomly initialize a neural network so that mean and variance of activity are uniform across neurons.Classically these statistics were defined over an ensemble of random networks.Alternatively, they can be 3mm defined over a random sample of inputs to the network.We show analytically and numerically that these two formulations of the principle of Pre-Natals mean-variance preservation are very different in deep networks using rectification nonlinearity (ReLU).We numerically investigate training speed after data-dependent initialization of networks to preserve sample mean and variance.