#### Sparsity-based Defense against Adversarial Attacks on Linear Classifiers

DNN image language MNIST speech

Deep neural networks represent the state of the art in machine learning in a growing number of fields, including vision, speech and natural language processing. However, recent work raises important questions about the robustness of such architectures, by showing that it is possible to induce class…

#### Non-Parametric Transformation Networks

ConvNets have been very effective in many applications where it is required to learn invariances to within-class nuisance transformations. However, through their architecture, ConvNets only enforce invariance to translation. In this paper, we introduce a new class of convolutional architectures cal…

#### Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

Convolutional Neural Network IMAGENET MNIST

Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that …

#### Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures

ads CIFAR Convolutional Neural Network MNIST

Major winning Convolutional Neural Networks (CNNs), such as AlexNet, VGGNet, ResNet, GoogleNet, include tens to hundreds of millions of parameters, which impose considerable computation and memory overhead. This limits their practical use for training, optimization and memory efficiency. On the con…

#### Design Exploration of Hybrid CMOS-OxRAM Deep Generative Architectures

Autoencoder Boltzmann Machine DNN MNIST

Deep Learning and its applications have gained tremendous interest recently in both academia and industry. Restricted Boltzmann Machines (RBMs) offer a key methodology to implement deep learning paradigms. This paper presents a novel approach for realizing hybrid CMOS-OxRAM based deep generative mo…

#### Introspective Classification with Convolutional Nets

Bayes CIFAR Convolutional Neural Network MNIST

We propose introspective convolutional networks (ICN) that emphasize the importance of having convolutional neural networks empowered with generative capabilities. We employ a reclassification-by-synthesis algorithm to perform training using a formulation stemmed from the Bayes theory. Our ICN trie…

#### TFGAN: A Lightweight Library for Generative Adversarial Networks

audio Generative Adversarial Network Google image MNIST video

Posted by Joel Shor, Senior Software Engineer, Machine Perception (Crossposted on the Google Open Source Blog) Training a neural network usually involves defining a loss function, which tells the network how close or far it is from its objective. For example, image classification networks are often…

#### Exploring the Space of Black-box Attacks on Deep Neural Networks

Existing black-box attacks on deep neural networks (DNNs) so far have largely focused on transferability, where an adversarial instance generated for a locally trained model can “transfer” to attack other learning models. In this paper, we propose novel Gradient Estimation black-box att…

#### Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy

We show that Entropy-SGD (Chaudhari et al., 2016), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by opt…

#### TIP: Typifying the Interpretability of Procedures

We provide a novel notion of what it means to be interpretable, looking past the usual association with human understanding. Our key insight is that interpretability is not an absolute concept and so we define it relative to a target model, which may or may not be a human. We define a framework tha…

#### The Robust Manifold Defense: Adversarial Training using Generative Models

DNN gradient image IMAGENET MNIST

Deep neural networks are demonstrating excellent performance on several classical vision problems. However, these networks are vulnerable to adversarial examples, minutely modified images that induce arbitrary attacker-chosen output from the network. We propose a mechanism to protect against these …

#### Mean Field Residual Networks: On the Edge of Chaos

We study randomly initialized residual networks using mean field theory and the theory of difference equations. Classical feedforward neural networks, such as those with tanh activations, exhibit exponential behavior on the average when propagating inputs forward or gradients backward. The exponent…

#### Learning in the Machine: Random Backpropagation and the Deep Learning Channel

Random backpropagation (RBP) is a variant of the backpropagation algorithm for training neural networks, where the transpose of the forward matrices are replaced by fixed random matrices in the calculation of the weight updates. It is remarkable both because of its effectiveness, in spite of using …

#### On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descent

gradient MNIST Reinforcement Learning

Because stochastic gradient descent (SGD) has shown promise optimizing neural networks with millions of parameters and few if any alternatives are known to exist, it has moved to the heart of leading approaches to reinforcement learning (RL). For that reason, the recent result from OpenAI showing t…

#### Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition

CIFAR DNN Gaussian Process Genetic Programming MNIST

We propose a method (TT-GP) for approximate inference in Gaussian Process (GP) models. We build on previous scalable GP research including stochastic variational inference based on inducing inputs, kernel interpolation, and structure exploiting algebra. The key idea of our method is to use Tensor T…

#### Meta-Learning via Feature-Label Memory Network

Deep learning typically requires training a very capable architecture using large datasets. However, many important learning problems demand an ability to draw valid inferences from small size datasets, and such problems pose a particular challenge for deep learning. In this regard, various researc…

#### Generalization in Deep Learning

This paper explains why deep learning can generalize well, despite large capacity and possible algorithmic instability, nonrobustness, and sharp minima, effectively addressing an open problem in the literature. Based on our theoretical insight, this paper also proposes a family of new regularizatio…

#### A systematic study of the class imbalance problem in convolutional neural networks

CIFAR Convolutional Neural Network DNN IMAGENET MNIST

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine l…

#### STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition

Spiking Neural Networks (SNNs) with a large number of weights and varied weight distribution can be difficult to implement in emerging in-memory computing hardware due to the limitations on crossbar size (implementing dot product), the constrained number of conductance levels in non-CMOS devices an…

#### Correlated Equilibria for Approximate Variational Inference in MRFs

Almost all of the work in graphical models for game theory has mirrored previous work in probabilistic graphical models. Our work considers the opposite direction: Taking advantage of recent advances in equilibrium computation for probabilistic inference. We present formulations of inference proble…

#### Projection Based Weight Normalization for Deep Neural Networks

CIFAR Convolutional Neural Network DNN gradient image IMAGENET MNIST

Optimizing deep neural networks (DNNs) often suffers from the ill-conditioned problem. We observe that the scaling-based weight space symmetry property in rectified nonlinear network will cause this negative effect. Therefore, we propose to constrain the incoming weights of each neuron to be unit-n…

#### DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks

audio clustering DNN image language MNIST security speech

Deep neural networks have become widely used, obtaining remarkable results in domains such as computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, and bio-informatics, where they have produced results comparable to human…

#### Neural Networks Regularization Through Class-wise Invariant Representation Learning

Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to s…

#### Improved Training for Self-Training

It is well known that for some tasks, labeled data sets may be hard to gather. Therefore, we wished to tackle here the problem of having insufficient training data. We examined learning methods from unlabeled data after an initial training on a limited labeled data set. The suggested approach can b…

#### Generative Adversarial Mapping Networks

CIFAR Generative Adversarial Network image MNIST

Generative Adversarial Networks (GANs) have shown impressive performance in generating photo-realistic images. They fit generative models by minimizing certain distance measure between the real image distribution and the generated data distribution. Several distance measures have been used, such as…