Lifelong Machine Learning Systems Beyond Learning Algorithms.pdf

上善若水

7页

0次

2022-03-24

免费下载

Lifelong Machine Learning Systems: Beyond Learning Algorithms

Daniel L. Silver

Jodrey School of Computer Science

Acadia University,

Wolfville, Nova Scotia, Canada B4P 2R6

Qiang Yang and Lianghao Li

Department of Computer Science and Engineering,

Hong Kong University of Science and Technology,

Clearwater Bay, Kowloon, Hong Kong

Abstract

Lifelong Machine Learning, or LML, considers sys-

tems that can learn many tasks from one or more do-

mains over its lifetime. The goal is to sequentially re-

tain learned knowledge and to selectively transfer that

knowledge when learning a new task so as to develop

more accurate hypotheses or policies. Following a re-

view of prior work on LML, we propose that it is

now appropriate for the AI community to move beyond

learning algorithms to more seriously consider the na-

ture of systems that are capable of learning over a life-

time. Reasons for our position are presented and poten-

tial counter-arguments are discussed. The remainder of

the paper contributes by deﬁning LML, presenting a ref-

erence framework that considers all forms of machine

learning, and listing several key challenges for and ben-

eﬁts from LML research. We conclude with ideas for

next steps to advance the ﬁeld.

Introduction

Over the last 25 years there have been signiﬁcant advances

in machine learning theory and algorithms. However, there

has been comparatively little work on systems that use these

algorithms to learn a variety of tasks over an extended period

of time such that the knowledge of the tasks is retained and

used to improve learning.

This position paper argues that it is now appropriate to

more seriously consider the nature of systems that are ca-

pable of learning, retaining and using knowledge over a life

time. In accord with (Thrun 1997), we call these lifelong

machine learning, or LML systems. We advocate that a sys-

tems approach is needed, taken in the context of an agent

that is able to acquire knowledge through learning, retain or

consolidate such knowledge, and use it for inductive transfer

when learning new tasks.

We argue that LML is a logical next step in machine learn-

ing research. The development and use of inductive bias

is essential to learning. There are a number of theoretical

advances in AI that will be found at the point where ma-

chine learning meets knowledge representation. There are

numerous practical applications of LML in areas such as

 2013, Association for the Advancement of Artiﬁcial

web agents and robotics. And our computing and communi-

cation systems now have the capacity to implement and test

LML systems.

This paper reviews prior work on LML that uses su-

pervised, unsupervised or reinforcement learning methods.

This work has gone by names such as constructive induc-

tion, incremental and continual learning, explanation-based

learning, sequential task learning, never ending learning,

and most recently learning with deep architectures. We then

present our position on the move beyond learning algorithms

to LML systems, detail the reasons for our position and dis-

cuss potential arguments and counter-arguments. We then

take some initial steps to advance LML research by propos-

ing a deﬁnition of LML and a reference framework for LML

that considers all forms of machine learning. We complete

the paper by listing several key challenges for and beneﬁts

from LML research and conclude with two ideas for advanc-

ing the ﬁeld.

Prior Work on LML

Their exists prior research in supervised, unsupervised and

reinforcement learning that consider systems that learn do-

mains of tasks over extended periods of time. In particular,

progress has been made in machine learning systems that ex-

hibit aspects of knowledge retention and inductive transfer.

Supervised Learning

In the mid 1980s Michalski introduced the theory of con-

structive inductive learning to cope with learning problems

in which the original representation space is inadequate for

the problem at hand (Michalski 1993). New knowledge is

hypothesized through two interrelated searches: (1) a search

for the best representational space for hypotheses and (2) a

search for the best hypothesis within the current represen-

tational space. The underlying principle is that new knowl-

edge is easier to induce if search is done using the right rep-

resentation.

In 1989 Solomonof began work on incremental learning

(Solomonoff 1989). His system was primed on a small, in-

complete set of primitive concepts, that are able to express

the solutions to the ﬁrst set of simple problems. When the

machine learns to use these concepts effectively it is given

more difﬁcult problems and, if necessary, additional primi-

tive concepts needed to solve them, and so on.

In the mid 1990s, Thrun and Mitchell worked on a

lifelong learning approached they called explanation-based

neural networks (Thrun 1996). EBNN is able to transfers

knowledge across multiple learning tasks. When faced with

a new learning task, EBNN exploits domain knowledge of

previous learning tasks (back-propagation gradients of prior

learned tasks) to guide the generalization of the new one. As

a result, EBNN generalizes more accurately from less data

than comparable methods. Thrun and Mitchell apply EBNN

transfer to autonomous robot learning when a multitude of

control learning tasks are encountered over an extended pe-

riod of time (Thrun and Mitchell 1995).

Since 1995, Silver et al. have proposed variants of se-

quential learning and consolidation systems using standard

back-propagation neural networks (Silver and Poirier 2004;

Silver, Poirier, and Currie 2008). A system of two multi-

ple task learning networks is used; one for short-term learn-

ing using task rehearsal to selectively transfer prior knowl-

edge, and a second for long-term consolidation using task

rehearsal to overcome the stability-plasticity problem. Task

rehearsal is an essential part of this system. After a task

has been successfully learned, its hypothesis representation

is saved. The saved hypothesis can be used to generate vir-

tual training examples so as to rehearse the prior task when

learning a new task. Knowledge is transferred to the new

task through the rehearsal of previously learned tasks within

the shared representation of the neural network. Similarly,

the knowledge of a new task can be consolidated into a large

domain knowledge network without loss of existing task

knowledge by using task rehearsal to maintain the function

accuracy of the prior tasks while the representation is modi-

ﬁed to accommodate the new task.

Rivest and Schultz proposed knowledge-based cascade-

correlation neural networks in the late 1990s (Shultz and

Rivest 2001). The method extends the original cascade-

correlation approach, by selecting previously learned sub-

networks as well as simple hidden units. In this way the

system is able to use past learning to bias new learning.

Unsupervised Learning

To overcome the stability-plasticity problem of forgetting

previous learned data clusters (concepts) Carpenter and

Grossberg proposed ART (Adaptive Resonance Theory)

neural networks (Grossberg 1987). Unsupervised ART net-

works learn a mapping between “bottom-up” input sensory

nodes and “top-down” expectation nodes (or cluster nodes).

The vector of new sensory data is compared with the vec-

tor of weights associated with one of the existing expecta-

tion nodes. If the difference does not exceed a set threshold,

called the “vigilance parameter”, the new example will be

considered a member of the most similar expectation node.

If the vigilance parameter is exceeded than a new expecta-

tion node is used and thus a new cluster is formed.

In (Strehl and Ghosh 2003), Trehl and Ghosh present a

cluster ensemble framework to reuse previous partitionings

of a set objects without accessing the original features. By

using the cluster label but not the original features, the pre-

existing knowledge can be reused to either create a single

consolidated cluster or generate a new partitioning of the

objects.

Raina et al. proposed the Self-taught Learning method

to build high-level features using unlabeled data for a set

of tasks (Raina et al. 2007). The authors used the features

to form a succinct input representation for future tasks and

achieve promising experimental results in several real appli-

cations such as image classiﬁcation, song genre classiﬁca-

tion and webpage classiﬁcation.

Carlson et al. (Carlson et al. 2010) describe the design and

partial implementation of a never-ending language learner,

or NELL, that each day must (1) extract, or read, informa-

tion from the web to populate a growing structured knowl-

edge base, and (2) learn to perform this task better than on

the previous day. The system uses a semi-supervised multi-

ple task learning approach in which a large number (531) of

different semantic functions are trained together in order to

improve learning accuracy.

Recent research into the learning of deep architectures of

neural networks can be connected to LML (Bengio 2009).

Layered neural networks of unsupervised Restricted Boltz-

man Machine and auto-encoders have been shown to efﬁ-

ciently develop hierarchies of features that capture regulari-

ties in their respective inputs. When used to learn a variety of

class categories, these networks develop layers of common

features similar to that seen in the visual cortex of humans.

Recently, Le et al. used the deep learning method to build

high-level features for large-scale applications by scaling up

the dataset, the model and the computational resources (Le

et al. 2012). By using millions of high resolution images

and very large neural networks, their system effectively dis-

cover high-level concepts like a cat’s face and a human body.

Experimental results on image classiﬁcation show that their

network can use its learned features to achieve a signiﬁ-

cant improvement in classiﬁcation performance over state-

of-the-art methods.

Reinforcement Learning

Several reinforcement learning researchers have considered

LML systems. In 1997, Ring proposed a lifelong learning

approach called continual learning that builds more compli-

cated skills on top of those already developed both incre-

mentally and hierarchically (Ring 1997). The system can

efﬁciently solve reinforcement-learning tasks and can then

transfer its skills to related but more complicated tasks.

Tanaka and Yamamura proposed a lifelong reinforcement

learning method for autonomous-robots by treating multi-

ple environments as multiple-tasks (Tanaka and Yamamura

1999). Parr and Russell used prior knowledge to reduce the

hypothesis space for reinforcement learning when the po-

lices considered by the learning process are constrained by

hierarchies (Parr and Russell 1997).

In (Sutton, Koop, and Silver 2007), Sutton et al. suggests

that learning should continue during an agent’s operations

since the environment may change making prior learning in-

sufﬁcient. In their work, an agent is proposed to adapt to dif-

ferent local environments when encountering different parts

of its world over an extended period of time. The experi-

mental results suggest continual tracking of a solution can