cnn_tutorial.pdf - 墨天轮文档

cnn_tutorial.pdf

Zixin Huo

8页

1次

2023-05-23

25墨值下载

Notes on Convolutional Neural Networks

Jake Bouvrie

Center for Biological and Computational Learning

Department of Brain and Cognitive Sciences

Massachusetts Institute of Technology

Cambridge, MA 02139

jvb@mit.edu

November 22, 2006

1 Introduction

This document discusses the derivation and implementation of convolutional neural networks

(CNNs) [3, 4], followed by a few straightforward extensions. Convolutional neural networks in-

volve many more connections than weights; the architecture itself realizes a form of regularization.

In addition, a convolutional network automatically provides some degree of translation invariance.

This particular kind of neural network assumes that we wish to learn ﬁlters, in a data-driven fash-

ion, as a means to extract features describing the inputs. The derivation we present is speciﬁc to

two-dimensional data and convolutions, but can be extended without much additional effort to an

arbitrary number of dimensions.

We begin with a description of classical backpropagation in fully connected networks, followed by a

derivation of the backpropagation updates for the ﬁltering and subsampling layers in a 2D convolu-

tional neural network. Throughout the discussion, we emphasize efﬁciency of the implementation,

and give small snippets of MATLAB code to accompany the equations. The importance of writing

efﬁcient code when it comes to CNNs cannot be overstated. We then turn to the topic of learning

how to combine feature maps from previous layers automatically, and consider in particular, learning

sparse combinations of feature maps.

Disclaimer: This rough note could contain errors, exaggerations, and false claims.

2 Vanilla Back-propagation Through Fully Connected Networks

In typical convolutional neural networks you might ﬁnd in the literature, the early analysis consists of

alternating convolution and sub-sampling operations, while the last stage of the architecture consists

of a generic multi-layer network: the last few layers (closest to the outputs) will be fully connected

1-dimensional layers. When you’re ready to pass the ﬁnal 2D feature maps as inputs to the fully

connected 1-D network, it is often convenient to just concatenate all the features present in all the

output maps into one long input vector, and we’re back to vanilla backpropagation. The standard

backprop algorithm will be described before going onto specializing the algorithm to the case of

convolutional networks (see e.g. [1] for more details).

2.1 Feedforward Pass

In the derivation that follows, we will consider the squared-error loss function. For a multiclass

problem with c classes and N training examples, this error is given by

n=1

k=1

− y

)

Here t

is the k-th dimension of the n-th pattern’s corresponding target (label), and y

is similarly

the value of the k-th output layer unit in response to the n-th input pattern. For multiclass classiﬁ-

cation problems, the targets will typically be organized as a “one-of-c” code where the k-th element

of t

is positive if the pattern x

belongs to class k. The rest of the entries of t

will be either zero

or negative depending on the choice of your output activation function (to be discussed below).

Because the error over the whole dataset is just a sum over the individual errors on each pattern, we

will consider backpropagation with respect to a single pattern, say the n-th one:

k=1

− y

)

− y

. (1)

With ordinary fully connected layers, we can compute the derivatives of E with respect to the net-

work weights using backpropagation rules of the following form. Let ` denote the current layer,

with the output layer designated to be layer L and the input “layer” designated to be layer 1. Deﬁne

the output of this layer to be

= f(u

), with u

= W

`−1

+ b

(2)

where the output activation function f (·) is commonly chosen to be the logistic (sigmoid) function

f(x) = (1 + e

−βx

)

−1

or the hyperbolic tangent function f(x) = a tanh(bx). The logistic function

maps [−∞, +∞] → [0, 1], while the hyperbolic tangent maps [−∞, +∞] → [−a, +a]. Therefore

while the outputs of the hyperbolic tangent function will typically be near zero, the outputs of a

sigmoid will be non-zero on average. However, normalizing your training data to have mean 0 and

variance 1 along the features can often improve convergence during gradient descent [5]. With a

normalized dataset, the hyperbolic tangent function is thus preferrable. LeCun recommends a =

1.7159 and b = 2/3, so that the point of maximum nonlinearity occurs at f(±1) = ±1 and will thus

avoid saturation during training if the desired training targets are normalized to take on the values

±1 [5].

2.2 Backpropagation Pass

The “errors” which we propagate backwards through the network can be thought of as “sensitivities”

of each unit with respect to perturbations of the bias

. That is to say,

∂E

∂b

∂E

∂u

∂b

= δ (3)

since in this case

∂u

∂b

= 1. So the bias sensitivity and the derivative of the error with respect to a

unit’s total input is equivalent. It is this derivative that is backpropagated from higher layers to lower

layers, using the following recurrence relation:

= (W

`+1

)

`+1

◦ f

) (4)

where “◦” denotes element-wise multiplication. For the error function (1), the sensitivities for the

output layer neurons will take a slightly different form:

= f

) ◦ (y

− t

Finally, the delta rule for updating a weight assigned to a given neuron is just a copy of the inputs

to that neuron, scaled by the neuron’s delta. In vector form, this is computed as an outer product

between the vector of inputs (which are the outputs from the previous layer) and the vector of

sensitivities:

∂E

∂W

= x

`−1

(δ

)

(5)

∆W

= −η

∂E

∂W

(6)

with analogous expressions for the bias update given by (3). In practice there is often a learning rate

parameter η

speciﬁc to each weight (W)

This nifty interpretation is due to Sebastian Seung