ICDE2024_AutoMC：Automated Model Compression based on Domain Knowledge and Progressive search strategy_腾讯云.pdf

迹部景吾

10页

0次

2025-04-22

免费下载

AutoMC: Automated Model Compression based on Domain Knowledge and

Progressive search strategy

Chunnan Wang, Hongzhi Wang, Xiangyu Shi

Harbin Institute of Technology

{WangChunnan,wangzh,xyu.shi}@hit.edu.cn

Abstract

Model compression methods can reduce model complex-

ity on the premise of maintaining acceptable performance,

and thus promote the application of deep neural networks

under resource constrained environments. Despite their

great success, the selection of suitable compression meth-

ods and design of details of the compression scheme are dif-

ﬁcult, requiring lots of domain knowledge as support, which

is not friendly to non-expert users. To make more users eas-

ily access to the model compression scheme that best meet

their needs, in this paper, we propose AutoMC, an effective

automatic tool for model compression. AutoMC builds the

domain knowledge on model compression to deeply under-

stand the characteristics and advantages of each compres-

sion method under different settings. In addition, it presents

a progressive search strategy to efﬁciently explore pareto

optimal compression scheme according to the learned prior

knowledge combined with the historical evaluation infor-

mation. Extensive experimental results show that AutoMC

can provide satisfying compression schemes within short

time, demonstrating the effectiveness of AutoMC.

1. Introduction

Neural networks are very powerful and can handle many

real-world tasks, but their parameter amounts are gener-

ally very large bring expensive computation and storage

cost. In order to apply them to mobile devices build-

ing more intelligent mobile devices, many model compres-

sion methods have been proposed, including model prun-

ing [2, 5, 8, 15, 21], knowledge distillation [27], low rank

approximation [2, 14] and so on.

These compression methods can effectively reduce

model parameters while maintaining model accuracy as

much as possible, but are difﬁcult to use. Each method has

many hyperparameters that can affect its compression ef-

fect, and different methods may suit for different compres-

sion tasks. Even the domain experts need lots of time to test

and analyze for designing a reasonable compression scheme

for a given compression task. This brings great challenges

to the practical application of compression techniques.

In order to enable ordinary users to easily and effec-

tively use the existing model compression techniques, in

this paper, we propose AutoMC, an Automatic Machine

Learning (AutoML) algorithm to help users automatically

design model compression schemes. Note that in AutoMC,

we do not limit a compression scheme to only use a com-

pression method under a speciﬁc setting. Instead, we allow

different compression methods and methods under different

hyperparameters settings to work together (execute sequen-

tially) to obtain diversiﬁed compression schemes. We try to

integrate advantages of different methods/settings through

this sequential combination so as to obtain more powerful

compression effect, and our ﬁnal experimental results prove

this idea to be effective and feasible.

However, the search space of AutoMC is huge. The

number of compression strategies

contained in the com-

pression scheme may be of any size, which brings great

challenges to the subsequent search tasks. In order to im-

prove the search efﬁciency, we present the following two in-

novations to improve the performance of AutoMC from the

perspectives of knowledge introduction and search space re-

duction, respectively.

Speciﬁcally, for the ﬁrst innovation, we built domain

knowledge on model compression, which discloses the

technical and settings details of compression strategies, and

their performance under some common compression tasks.

This domain knowledge can assist AutoMC to deeply un-

derstand the potential characteristics and advantages of each

component in the search space. It can guide AutoMC select

more appropriate compression strategies to build effective

compression schemes, and thus reduce useless evaluation

and improve the search efﬁciency.

As for the second innovation, we adopted the idea of pro-

gressive search space expansion to improve the search efﬁ-

ciency of AutoMC. Speciﬁcally, in each round of optimiza-

In this paper, a compression strategy refers to a compression method

with a speciﬁc hyperparameter setting.

arXiv:2201.09884v1 [cs.LG] 24 Jan 2022

tion, we only take the next operations, i.e., unexplored next-

step compression strategies, of the evaluated compression

scheme as the search space. Then, we select the pareto op-

timal operations for scheme evaluation, and ﬁnally take the

next operations of the new scheme as the newly expanded

search area to participate in the next round of optimization.

In this way, AutoMC can selectively and gradually explore

more valuable search space, reduce the search difﬁculty, and

improve the search efﬁciency. In addition, AutoMC can

analyze and compare the impact of subsequent operations

on the performance of each compression scheme in a ﬁne-

grained manner, and ﬁnalize a more valuable next-step ex-

ploration route for implementation, thereby effectively re-

ducing the evaluation of useless schemes.

The ﬁnal experimental results show that AutoMC can

quickly search for powerful model compression schemes.

Compared with the existing AutoML algorithms which are

non-progressive and ignore domain knowledge, AutoMC is

more suitable for dealing with the automatic model com-

pression problem where search space is huge and compo-

nents are complete and executable algorithms.

Our contributions are summarized as follows:

1. Automation. AutoMC can automatically design the

effective model compression scheme according to the

user demands. As far as we know, this is the ﬁrst auto-

matic model compression tool.

2. Innovation. In order to improve the search efﬁciency

of AutoMC algorithm, an effective analysis method

based on domain knowledge and a progressive search

strategy are designed. As far as we know, AutoMC

is the ﬁrst AutoML algorithm that introduce external

knowledge.

3. Effectiveness. Extensive experimental results show

that with the help of domain knowledge and progres-

sive search strategy, AutoMC can efﬁciently search the

optimal model compression scheme for users, outper-

forming compression methods designed by humans.

2. Related Work

2.1. Model Compression Methods

Model compression is the key point of applying neural

networks to mobile or embedding devices, and has been

widely studied all over the world. Researchers have pro-

posed many effective compression methods, and they can

be roughly divided into the following four categories. (1)

pruning methods, which aim to remove redundant parts

e.g., ﬁlters, channels, kernels or layers, from the neural

network [7, 17, 18, 22]; (2) knowledge distillation methods

that train the compact and computationally efﬁcient neural

model with the supervision from well-trained larger models;

(3) low-rank approximation methods that split the convolu-

tional matrices into small ones using decomposition tech-

niques [16]; (4) quantization methods that reduce the preci-

sion of parameter values of the neural network [10, 29].

These compression methods have their own advantages,

and have achieved great success in many compression tasks,

but are difﬁcult to apply as is discussed in the introduction

part. In this paper, we aim to ﬂexibly use the experience

provided by them to support the automatic design of model

compression schemes.

2.2. Automated Machine Learning Algorithms

The goal of Auto

mated Machine Learning (AutoML) is

to realize the progressive automation of ML, including au-

tomatic design of neural network architecture, ML work-

ﬂow [9,28] and automatic setting of hyperparameters of ML

model [11,23]. The idea of the existing AutoML algorithms

is to deﬁne an effective search space which contains a va-

riety of solutions, then design an efﬁcient search strategy

to quickly ﬁnd the best ML solution from the search space,

and ﬁnally take the best solution as the ﬁnal output.

Search strategy has a great impact on the performance

of the AutoML algorithm. The existing AutoML search

strategies can be divided into 3 categories: Reinforcement

Learning (RL) methods [1], Evolutionary Algorithm (EA)

based methods [4, 25] and gradient-based methods [20, 24].

The RL-based methods use a recurrent network as con-

troller to determine a sequence of operators, thus construct

the ML solution sequentially. EA-based methods initialize a

population of ML solutions ﬁrst and then evolve them with

their validation accuracies as ﬁtnesses. As for the gradient-

based methods, they are designed for neural architecture

search problems. They relax the search space to be contin-

uous, so that the architecture can be optimized with respect

to its validation performance by gradient descent [3]. They

fail to deal with the search space composed of executable

compression strategies. Therefore, we only compare Au-

toMC’s search strategy with the previous two methods.

3. Our Approach

We ﬁrstly give the related concepts on model compres-

sion and problem deﬁnition of automatic model compres-

sion (Section 3.1). Then, we make full use of the exist-

ing experience to construct an efﬁcient search space for

the compression area (Section 3.2). Finally, we designed

a search strategy, which improves the search efﬁciency

from the perspectives of knowledge introduction and search

space reduction, to help users quickly search for the optimal

compression scheme (Section 3.3).

3.1. Related Concepts and Problem Deﬁnition

Related Concepts. Given a neural model M , we use

P (M ), F (M) and A(M ) to denote its parameter amount,

of 10

免费下载

关注

评论