Projection Based Weight Normalization for Deep Neural Networks
This topic contains 0 replies, has 1 voice, and was last updated by arXiv 1 year, 4 months ago.

Projection Based Weight Normalization for Deep Neural Networks
Optimizing deep neural networks (DNNs) often suffers from the illconditioned problem. We observe that the scalingbased weight space symmetry property in rectified nonlinear network will cause this negative effect. Therefore, we propose to constrain the incoming weights of each neuron to be unitnorm, which is formulated as an optimization problem over Oblique manifold. A simple yet efficient method referred to as projection based weight normalization (PBWN) is also developed to solve this problem. PBWN executes standard gradient updates, followed by projecting the updated weight back to Oblique manifold. This proposed method has the property of regularization and collaborates well with the commonly used batch normalization technique. We conduct comprehensive experiments on several widelyused image datasets including CIFAR10, CIFAR100, SVHN and ImageNet for supervised learning over the stateoftheart convolutional neural networks, such as Inception, VGG and residual networks. The results show that our method is able to improve the performance of DNNs with different architectures consistently. We also apply our method to Ladder network for semisupervised learning on permutation invariant MNIST dataset, and our method outperforms the stateoftheart methods: we obtain test errors as 2.52%, 1.06%, and 0.91% with only 20, 50, and 100 labeled samples, respectively.
Projection Based Weight Normalization for Deep Neural Networks
by Lei Huang, Xianglong Liu, Bo Lang, Bo Li
https://arxiv.org/pdf/1710.02338v1.pdf
You must be logged in to reply to this topic.