2017CVPR最佳论文-密集连接的卷积网络.pdf

poPoq

10页

0次

2021-05-01

50墨值下载

Densely Connected Convolutional Networks

Gao Huang

∗

Cornell University

gh349@cornell.edu

Zhuang Liu

∗

Tsinghua University

liuzhuang13@mails.tsinghua.edu.cn

Laurens van der Maaten

Facebook AI Research

lvdmaaten@fb.com

Kilian Q. Weinberger

Cornell University

kqw4@cornell.edu

Abstract

Recent work has shown that convolutional networks can

be substantially deeper, more accurate, and efﬁcient to train

if they contain shorter connections between layers close to

the input and those close to the output. In this paper, we

embrace this observation and introduce the Dense Convo-

lutional Network (DenseNet), which connects each layer

to every other layer in a feed-forward fashion. Whereas

traditional convolutional networks with L layers have L

connections—one between each layer and its subsequent

layer—our network has

L(L+1)

direct connections. For

each layer, the feature-maps of all preceding layers are

used as inputs, and its own feature-maps are used as inputs

into all subsequent layers. DenseNets have several com-

pelling advantages: they alleviate the vanishing-gradient

problem, strengthen feature propagation, encourage fea-

ture reuse, and substantially reduce the number of parame-

ters. We evaluate our proposed architecture on four highly

competitive object recognition benchmark tasks (CIFAR-10,

CIFAR-100, SVHN, and ImageNet). DenseNets obtain sig-

niﬁcant improvements over the state-of-the-art on most of

them, whilst requiring less computation to achieve high per-

formance. Code and pre-trained models are available at

https://github.com/liuzhuang13/DenseNet.

1. Introduction

Convolutional neural networks (CNNs) have become

the dominant machine learning approach for visual object

recognition. Although they were originally introduced over

20 years ago [

18], improvements in computer hardware and

network structure have enabled the training of truly deep

CNNs only recently. The original LeNet5 [

19] consisted of

5 layers, VGG featured 19 [

28], and only last year Highway

∗

Authors contributed equally

Figure 1: A 5-layer dense block with a growth rate of k = 4.

Each layer takes all preceding feature-maps as input.

Networks [33] and Residual Networks (ResNets) [11] have

surpassed the 100-layer barrier.

As CNNs become increasingly deep, a new research

problem emerges: as information about the input or gra-

dient passes through many layers, it can vanish and “wash

out” by the time it reaches the end (or beginning) of the

network. Many recent publications address this or related

problems. ResNets [

11] and Highway Networks [33] by-

pass signal from one layer to the next via identity connec-

tions. Stochastic depth [13] shortens ResNets by randomly

dropping layers during training to allow better information

and gradient ﬂow. FractalNets [

17] repeatedly combine sev-

eral parallel layer sequences with different number of con-

volutional blocks to obtain a large nominal depth, while

maintaining many short paths in the network. Although

these different approaches vary in network topology and

training procedure, they all share a key characteristic: they

create short paths from early layers to later layers.

4700

In this paper, we propose an architecture that distills this

insight into a simple connectivity pattern: to ensure maxi-

mum information ﬂow between layers in the network, we

connect all layers (with matching feature-map sizes) di-

rectly with each other. To preserve the feed-forward nature,

each layer obtains additional inputs from all preceding lay-

ers and passes on its own feature-maps to all subsequent

layers. Figure

1 illustrates this layout schematically. Cru-

cially, in contrast to ResNets, we never combine features

through summation before they are passed into a layer; in-

stead, we combine features by concatenating them. Hence,

the ℓ

layer has ℓ inputs, consisting of the feature-maps

of all preceding convolutional blocks. Its own feature-maps

are passed on to all L−ℓ subsequent layers. This introduces

L(L+1)

connections in an L-layer network, instead of just

L, as in traditional architectures. Because of its dense con-

nectivity pattern, we refer to our approach as Dense Convo-

lutional Network (DenseNet).

A possibly counter-intuitive effect of this dense connec-

tivity pattern is that it requires fewer parameters than tra-

ditional convolutional networks, as there is no need to re-

learn redundant feature-maps. Traditional feed-forward ar-

chitectures can be viewed as algorithms with a state, which

is passed on from layer to layer. Each layer reads the state

from its preceding layer and writes to the subsequent layer.

It changes the state but also passes on information that needs

to be preserved. ResNets [

11] make this information preser-

vation explicit through additive identity transformations.

Recent variations of ResNets [13] show that many layers

contribute very little and can in fact be randomly dropped

during training. This makes the state of ResNets similar

to (unrolled) recurrent neural networks [

21], but the num-

ber of parameters of ResNets is substantially larger because

each layer has its own weights. Our proposed DenseNet ar-

chitecture explicitly differentiates between information that

is added to the network and information that is preserved.

DenseNet layers are very narrow (e.g., 12 ﬁlters per layer),

adding only a small set of feature-maps to the “collective

knowledge” of the network and keep the remaining feature-

maps unchanged—and the ﬁnal classiﬁer makes a decision

based on all feature-maps in the network.

Besides better parameter efﬁciency, one big advantage of

DenseNets is their improved ﬂow of information and gra-

dients throughout the network, which makes them easy to

train. Each layer has direct access to the gradients from the

loss function and the original input signal, leading to an im-

plicit deep supervision [20]. This helps training of deeper

network architectures. Further, we also observe that dense

connections have a regularizing effect, which reduces over-

ﬁtting on tasks with smaller training set sizes.

We evaluate DenseNets on four highly competitive

benchmark datasets (CIFAR-10, CIFAR-100, SVHN, and

ImageNet). Our models tend to require much fewer param-

eters than existing algorithms with comparable accuracy.

Further, we signiﬁcantly outperform the current state-of-

the-art results on most of the benchmark tasks.

2. Related Work

The exploration of network architectures has been a part

of neural network research since their initial discovery. The

recent resurgence in popularity of neural networks has also

revived this research domain. The increasing number of lay-

ers in modern networks ampliﬁes the differences between

architectures and motivates the exploration of different con-

nectivity patterns and the revisiting of old research ideas.

A cascade structure similar to our proposed dense net-

work layout has already been studied in the neural networks

literature in the 1980s [

3]. Their pioneering work focuses on

fully connected multi-layer perceptrons trained in a layer-

by-layer fashion. More recently, fully connected cascade

networks to be trained with batch gradient descent were

proposed [

39]. Although effective on small datasets, this

approach only scales to networks with a few hundred pa-

rameters. In [

9, 23, 30, 40], utilizing multi-level features

in CNNs through skip-connnections has been found to be

effective for various vision tasks. Parallel to our work, [

derived a purely theoretical framework for networks with

cross-layer connections similar to ours.

Highway Networks [

33] were amongst the ﬁrst architec-

tures that provided a means to effectively train end-to-end

networks with more than 100 layers. Using bypassing paths

along with gating units, Highway Networks with hundreds

of layers can be optimized without difﬁculty. The bypass-

ing paths are presumed to be the key factor that eases the

training of these very deep networks. This point is further

supported by ResNets [

11], in which pure identity mappings

are used as bypassing paths. ResNets have achieved im-

pressive, record-breaking performance on many challeng-

ing image recognition, localization, and detection tasks,

such as ImageNet and COCO object detection [

11]. Re-

cently, stochastic depth was proposed as a way to success-

fully train a 1202-layer ResNet [13]. Stochastic depth im-

proves the training of deep residual networks by dropping

layers randomly during training. This shows that not all

layers may be needed and highlights that there is a great

amount of redundancy in deep (residual) networks. Our pa-

per was partly inspired by that observation. ResNets with

pre-activation also facilitate the training of state-of-the-art

networks with > 1000 layers [

12].

An orthogonal approach to making networks deeper

(e.g., with the help of skip connections) is to increase the

network width. The GoogLeNet [

35, 36] uses an “Incep-

tion module” which concatenates feature-maps produced

by ﬁlters of different sizes. In [

37], a variant of ResNets

with wide generalized residual blocks was proposed. In

fact, simply increasing the number of ﬁlters in each layer of

4701

of 10

50墨值下载

自动驾驶

关注

评论