Net2net_ Accelerating learning via knowledge transfer.pdf

上善若水

320

12页

0次

2022-03-24

免费下载

Published as a conference paper at ICLR 2016

Net2Net: ACCELERATING LEARNING

VIA KNOWLEDGE TRANSFER

Tianqi Chen

∗

, Ian Goodfellow, and Jonathon Shlens

Google Inc., Mountain View, CA

tqchen@cs.washington.edu, {goodfellow,shlens}@google.com

ABSTRACT

We introduce techniques for rapidly transferring the information stored in one

neural net into another neural net. The main purpose is to accelerate the train-

ing of a signiﬁcantly larger neural net. During real-world workﬂows, one often

trains very many different neural networks during the experimentation and de-

sign process. This is a wasteful process in which each new model is trained from

scratch. Our Net2Net technique accelerates the experimentation process by in-

stantaneously transferring the knowledge from a previous network to each new

deeper or wider network. Our techniques are based on the concept of function-

preserving transformations between neural network speciﬁcations. This differs

from previous approaches to pre-training that altered the function represented by

a neural net when adding layers to it. Using our knowledge transfer mechanism

to add depth to Inception modules, we demonstrate a new state of the art accuracy

rating on the ImageNet dataset.

1 INTRODUCTION

We propose a new kind of operation to perform on large neural networks: rapidly transfering knowl-

edge contained in one neural network to another neural network. We call this the Net2Net fam-

ily of operations. We use Net2Net as a general term describing any process of training a stu-

dent network signiﬁcantly faster than would otherwise be possible by leveraging knowledge from a

teacher network that was already trained on the same task. In this article, we propose two speciﬁc

Net2Net methodologies. Both are based on the idea of function-preserving transformations of

neural networks. Speciﬁcally, we initialize the student to be a neural network that represents the

same function as the teacher, but using a different parameterization. One of these transformations,

Net2WiderNet allows replacing a model with an equivalent model that is wider (has more units in

each hidden layer). Another of these transformations, Net2DeeperNet allows replacing a model

that satisﬁes some properties with an equivalent, deeper model. After initializing the larger network

to contain all of the knowledge previously acquired by the smaller network, the larger network may

be trained to improve its performance.

Traditionally, machine learning algorithms have been designed to receive a ﬁxed dataset as input,

initialize a new model with no knowledge, and train that model to convergence on that dataset. Real

workﬂows are considerably more complicated than this idealized scenario. We advocate Net2Net

operations as a useful tool for accelerating real-world workﬂows.

One way that real workﬂows deviate from the idealized scenario is that machine learning practition-

ers usually do not train only a single model on each dataset. Instead, one typically trains multiple

models, with each model designed to improve upon the previous model in some way. Each step in

the iterative design process relies on fully training and evaluating the innovation from the previous

step. For many large models, training is a long process, lasting for a week or even for a month. This

makes data-driven iterative design slow, due to the latency of evaluating whether each change to the

model caused an improvement.

Net2Net operations accelerate these workﬂows by rapidly transferring knowledge from the pre-

vious best model into each new model that an experimenter proposes. Instead of training each

considered design of model for as much as a month, the experimenter can use Net2Net to train the

∗

Tianqi Chen is also a PhD student at University of Washington.

arXiv:1511.05641v4 [cs.LG] 23 Apr 2016

Published as a conference paper at ICLR 2016

Figure 1: Comparison between a traditional workﬂow and the Net2Net Workﬂow; Net2Net

reuses information from an already trained model to speed up the training of a new model.

model for a shorter period of time beginning from the function learned by the previous best model.

Fig 1 demonstrates the difference of this approach from traditional one.

More ambitiously, real machine learning systems will eventually become lifelong learning sys-

tems (Thrun, 1995; Silver et al., 2013; Mitchell et al., 2015). These machine learning systems need

to continue to function for long periods of time and continually experience new training examples

as these examples become available. We can think of a lifelong learning system as experiencing a

continually growing training set. The optimal model complexity changes as training set size changes

over time. Initially, a small model may be preferred, in order to prevent overﬁtting and to reduce

the computational cost of using the model. Later, a large model may be necessary to fully utilize

the large dataset. Net2Net operations allow us to smoothly instantiate a signiﬁcantly larger model

and immediately begin using it in our lifelong learning system, rather than needing to spend weeks

or months re-train a larger model from scratch on the latest, largest version of the training set.

2 METHODOLOGY

In this section, we describe our new Net2Net operations and how we applied them on real deep

neural nets.

2.1 FEATURE PREDICTION

We brieﬂy experimented with a method that proved not to offer a signiﬁcant advantage: training a

large student network beginning from a random initialization, and introducing a set of extra “teacher

prediction” layers into the student network. Speciﬁcally, several convolutional hidden layers of the

student network were provided as input to new, learned, convolutional layers. The cost function

was modiﬁed to include terms encouraging the output of these auxiliary layers to be close to a

corresponding layer in the teacher network. In other words, the student is trained to use each of its

hidden layers to predict the values of the hidden layers in the teacher.

The goal was that the teacher would provide a good internal representation for the task that the stu-

dent could quickly copy and then begin to reﬁne. The approach resembles the FitNets (Romero et al.,

2014) strategy for training very thin networks of moderate depth. Unfortunately, we did not ﬁnd that

this method offered any compelling speedup or other advantage relative to the baseline approach.

This may be because our baseline was very strong, based on training with batch normalization (Ioffe

& Szegedy, 2015). Mahayri et al. (2015) independently observed that the beneﬁts of the FitNets

training strategy were eliminated after changing the model to use batch normalization.

The FitNets-style approach to Net2Net learning is very general, in the sense that, if successful, it

would allow any architecture of student network to learn from any architecture of teacher network.

Though we were not able to make this general approach work, we encourage other researchers to

attempt to design fully general Net2Net strategies in the future. We instead turned to different

Net2Net strategies that were limited in scope but more effective.

2.2 FUNCTION-PRESERVING INITIALIZATIONS

We introduce two effective Net2Net strategies. Both are based on initializing the student network

to represent the same function as the teacher, then continuing to train the student network by normal

means. Speciﬁcally, suppose that a teacher network is represented by a function y = f(x; θ) where

of 12

免费下载

深度学习 paper

关注

评论