disk_failures.pdf - 墨天轮文档

disk_failures.pdf

逆风飞翔

218

13页

0次

2022-06-14

免费下载

Appears in the Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST’07), February 2007

Failure Trends in a Large Disk Drive Population

Eduardo Pinheiro, Wolf-Dietrich Weber and Luiz Andr

e Barroso

Google Inc.

1600 Amphitheatre Pkwy

Mountain View, CA 94043

{edpin,wolf,luiz}@google.com

Abstract

It is estimated that over 90% of all new information produced

in the world is being stored on magnetic media, most of it on

hard disk drives. Despite their importance, there is relatively

little published work on the failure patterns of disk drives, and

the key factors that affect their lifetime. Most available data

are either based on extrapolation from accelerated aging exper-

iments or from relatively modest sized ﬁeld studies. Moreover,

larger population studies rarely have the infrastructure in place

to collect health signals from components in operation, which

is critical information for detailed failure analysis.

We present data collected from detailed observations of a

large disk drive population in a production Internet services de-

ployment. The population observed is many times larger than

that of previous studies. In addition to presenting failure statis-

tics, we analyze the correlation between failures and several

parameters generally believed to impact longevity.

Our analysis identiﬁes several parameters from the drive’s

self monitoring facility (SMART) that correlate highly with

failures. Despite this high correlation, we conclude that mod-

els based on SMART parameters alone are unlikely to be useful

for predicting individual drive failures. Surprisingly, we found

that temperature and activity levels were much less correlated

with drive failures than previously reported.

1 Introduction

The tremendous advances in low-cost, high-capacity

magnetic disk drives have been among the key factors

helping establish a modern society that is deeply reliant

on information technology. High-volume, consumer-

grade disk drives have become such a successful prod-

uct that their deployments range from home computers

and appliances to large-scale server farms. In 2002, for

example, it was estimated that over 90% of all new in-

formation produced was stored on magnetic media, most

of it being hard disk drives [12]. It is therefore critical

to improve our understanding of how robust these com-

ponents are and what main factors are associated with

failures. Such understanding can be particularly useful

for guiding the design of storage systems as well as de-

vising deployment and maintenance strategies.

Despite the importance of the subject, there are very

few published studies on failure characteristics of disk

drives. Most of the available information comes from

the disk manufacturers themselves [2]. Their data are

typically based on extrapolation from accelerated life

test data of small populations or from returned unit

databases. Accelerated life tests, although useful in pro-

viding insight into how some environmental factors can

affect disk drive lifetime, have been known to be poor

predictors of actual failure rates as seen by customers

in the ﬁeld [7]. Statistics from returned units are typi-

cally based on much larger populations, but since there

is little or no visibility into the deployment characteris-

tics, the analysis lacks valuable insight into what actu-

ally happened to the drive during operation. In addition,

since units are typically returned during the warranty pe-

riod (often three years or less), manufacturers’ databases

may not be as helpful for the study of long-term effects.

A few recent studies have shed some light on ﬁeld

failure behavior of disk drives [6, 7, 9, 16, 17, 19, 20].

However, these studies have either reported on relatively

modest populations or did not monitor the disks closely

enough during deployment to provide insights into the

factors that might be associated with failures.

Disk drives are generally very reliable but they are

also very complex components. This combination

means that although they fail rarely, when they do fail,

the possible causes of failure can be numerous. As a

result, detailed studies of very large populations are the

only way to collect enough failure statistics to enable

meaningful conclusions. In this paper we present one

such study by examining the population of hard drives

under deployment within Google’s computing infras-

tructure.

We have built an infrastructure that collects vital in-

formation about all Google’s systems every few min-

utes, and a repository that stores these data in time-

series format (essentially forever) for further analysis.

The information collected includes environmental fac-

tors (such as temperatures), activity levels and many of

the Self-Monitoring Analysis and Reporting Technology

(SMART) parameters that are believed to be good indi-

cators of disk drive health. We mine through these data

and attempt to ﬁnd evidence that corroborates or con-

tradicts many of the commonly held beliefs about how

various factors can affect disk drive lifetime.

Our paper is unique in that it is based on data from a

disk population size that is typically only available from

vendor warranty databases, but has the depth of deploy-

ment visibility and detailed lifetime follow-up that only

an end-user study can provide. Our key ﬁndings are:

• Contrary to previously reported results, we found

very little correlation between failure rates and ei-

ther elevated temperature or activity levels.

• Some SMART parameters (scan errors, realloca-

tion counts, ofﬂine reallocation counts, and proba-

tional counts) have a large impact on failure proba-

bility.

• Given the lack of occurrence of predictive SMART

signals on a large fraction of failed drives, it is un-

likely that an accurate predictive failure model can

be built based on these signals alone.

2 Background

In this section we describe the infrastructure that was

used to gather and process the data used in this study,

the types of disk drives included in the analysis, and in-

formation on how they are deployed.

2.1 The System Health Infrastructure

The System Health infrastructure is a large distributed

software system that collects and stores hundreds of

attribute-value pairs from all of Google’s servers, and

provides the interface for arbitrary analysis jobs to pro-

cess that data.

The architecture of the System Health infrastructure

is shown in Figure 1. It consists of a data collection

layer, a distributed repository and an analysis frame-

work. The collection layer is responsible for getting in-

formation from each of thousands of individual servers

into a centralized repository. Different ﬂavors of col-

lectors exist to gather different types of data. Much of

the health information is obtained from the machines di-

rectly. A daemon runs on every machine and gathers

local data related to that machine’s health, such as envi-

ronmental parameters, utilization information of various

Figure 1: Collection, storage, and analysis architecture.

resources, error indications, and conﬁguration informa-

tion. It is imperative that this daemon’s resource usage

be very light, so not to interfere with the applications.

One way to assure this is to have the machine-level col-

lector poll individual machines relatively infrequently

(every few minutes). Other slower changing data (such

as conﬁguration information) and data from other exist-

ing databases can be collected even less frequently than

that. Most notably for this study, data regarding ma-

chine repairs and disk swaps are pulled in from another

database.

The System Health database is built upon Bigtable

[3], a distributed data repository widely used within

Google, which itself is built upon the Google File Sys-

tem (GFS) [8]. Bigtable takes care of all the data layout,

compression, and access chores associated with a large

data store. It presents the abstraction of a 2-dimensional

table of data cells, with different versions over time mak-

ing up a third dimension. It is a natural ﬁt for keeping

track of the values of different variables (columns) for

different machines (rows) over time. The System Health

database thus retains a complete time-ordered history of

the environment, utilization, error, conﬁguration, and re-

pair events in each machine’s life.

Analysis programs run on top of the System Health

database, looking at information from individual ma-

chines, or mining the data across thousands of machines.

Large-scale analysis programs are typically built upon

Google’s Mapreduce [5] framework. Mapreduce auto-

mates the mechanisms of large-scale distributed compu-

of 13

免费下载

关注

评论