The Google File System.pdf
免费下载
The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
Google
∗
ABSTRACT
We have designed and implemented the Google File Sys-
tem, a scalable distributed file system for large distributed
data-intensive applications. It provides fault tolerance while
running on inexpensive commodity hardware, and it delivers
high aggregate performance to a large number of clients.
While sharing many of the same goals as previous dis-
tributed file systems, our design has been driven by obser-
vations of our application workloads and technological envi-
ronment, both current and anticipated, that reflect a marked
departure from some earlier file system assumptions. This
has led us to reexamine traditional choices and explore rad-
ically different design points.
The file system has successfully met our storage needs.
It is widely deployed within Google as the storage platform
for the generation and processing of data used by our ser-
vice as well as research and development efforts that require
large data sets. The largest cluster to date provides hun-
dreds of terabytes of storage across thousands of disks on
over a thousand machines, and it is concurrently accessed
by hundreds of clients.
In this paper, we present file system interface extensions
designed to support distributed applications, discuss many
aspects of our design, and report measurements from both
micro-benchmarks and real world use.
Categories and Subject Descriptors
D[4]: 3—Distributed file systems
General Terms
Design, reliability, performance, measurement
Keywords
Fault tolerance, scalability, data storage, clustered storage
∗
The authors can be reached at the following addresses:
{sanjay,hgobioff,shuntak}@google.com.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
SOSP’03, October 19–22, 2003, Bolton Landing, New York, USA.
Copyright 2003 ACM 1-58113-757-5/03/0010 ...
$5.00.
1. INTRODUCTION
We have designed and implemented the Google File Sys-
tem (GFS) to meet the rapidly growing demands of Google’s
data processing needs. GFS shares many of the same goals
as previous distributed file systems such as performance,
scalability, reliability, and availability. However, its design
has been driven by key observations of our application work-
loads and technological environment, both current and an-
ticipated, that reflect a marked departure from some earlier
file system design assumptions. We have reexamined tradi-
tional choices and explored radically different points in the
design space.
First, component failures are the norm rather than the
exception. The file system consists of hundreds or even
thousands of storage machines built from inexpensive com-
modity parts and is accessed by a comparable number of
client machines. The quantity and quality of the compo-
nents virtually guarantee that some are not functional at
any given time and some will not recover from their cur-
rent failures. We have seen problems caused by application
bugs, operating system bugs, human errors, and the failures
of disks, memory, connectors, networking, and power sup-
plies. Therefore, constant monitoring, error detection, fault
tolerance, and automatic recovery must be integral to the
system.
Second, files are huge by traditional standards. Multi-GB
files are common. Each file typically contains many applica-
tion objects such as web documents. When we are regularly
working with fast growing data sets of many TBs comprising
billions of objects, it is unwieldy to manage billions of ap-
proximately KB-sized files even when the file system could
support it. As a result, design assumptions and parameters
such as I/O operation and block sizes have to be revisited.
Third, most files are mutated by appending new data
rather than overwriting existing data. Random writes within
a file are practically non-existent. Once written, the files
are only read, and often only sequentially. A variety of
data share these characteristics. Some may constitute large
repositories that data analysis programs scan through. Some
may be data streams continuously generated by running ap-
plications. Some may be archival data. Some may be in-
termediate results produced on one machine and processed
on another, whether simultaneously or later in time. Given
this access pattern on huge files, appending becomes the fo-
cus of performance optimization and atomicity guarantees,
while caching data blocks in the client loses its appeal.
Fourth, co-designing the applications and the file system
API benefits the overall system by increasing our flexibility.
For example, we have relaxed GFS’s consistency model to
vastly simplify the file system without imposing an onerous
burden on the applications. We have also introduced an
atomic append operation so that multiple clients can append
concurrently to a file without extra synchronization between
them. These will be discussed in more details later in the
paper.
Multiple GFS clusters are currently deployed for different
purposes. The largest ones have over 1000 storage nodes,
over 300 TB of disk storage, and are heavily accessed by
hundreds of clients on distinct machines on a continuous
basis.
2. DESIGN OVERVIEW
2.1 Assumptions
In designing a file system for our needs, we have been
guided by assumptions that offer both challenges and op-
portunities. We alluded to some key observations earlier
and now lay out our assumptions in more details.
• The system is built from many inexpensive commodity
components that often fail. It must constantly monitor
itself and detect, tolerate, and recover promptly from
component failures on a routine basis.
• The system stores a modest number of large files. We
expect a few million files, each typically 100 MB or
larger in size. Multi-GB files are the common case
and should be managed efficiently. Small files must be
supported, but we need not optimize for them.
• The workloads primarily consist of two kinds of reads:
large streaming reads and small random reads. In
large streaming reads, individual operations typically
read hundreds of KBs, more commonly 1 MB or more.
Successive operations from the same client often read
through a contiguous region of a file. A small ran-
dom read typically reads a few KBs at some arbitrary
offset. Performance-conscious applications often batch
and sort their small reads to advance steadily through
the file rather than go back and forth.
• The workloads also have many large, sequential writes
that append data to files. Typical operation sizes are
similar to those for reads. Once written, files are sel-
dom modified again. Small writes at arbitrary posi-
tions in a file are supported but do not have to be
efficient.
• The system must efficiently implement well-defined se-
mantics for multiple clients that concurrently append
to the same file. Our files are often used as producer-
consumer queues or for many-way merging. Hundreds
of producers, running one per machine, will concur-
rently append to a file. Atomicity with minimal syn-
chronization overhead is essential. The file may be
read later, or a consumer may be reading through the
file simultaneously.
• High sustained bandwidth is more important than low
latency. Most of our target applications place a pre-
mium on processing data in bulk at a high rate, while
few have stringent response time requirements for an
individual read or write.
2.2 Interface
GFS provides a familiar file system interface, though it
does not implement a standard API such as POSIX. Files are
organized hierarchically in directories and identified by path-
names. We support the usual operations to create, delete,
open, close, read,andwrite files.
Moreover, GFS has snapshot and record append opera-
tions. Snapshot creates a copy of a file or a directory tree
at low cost. Record append allows multiple clients to ap-
pend data to the same file concurrently while guaranteeing
the atomicity of each individual client’s append. It is use-
ful for implementing multi-way merge results and producer-
consumer queues that many clients can simultaneously ap-
pend to without additional locking. We have found these
types of files to be invaluable in building large distributed
applications. Snapshot and record append are discussed fur-
ther in Sections 3.4 and 3.3 respectively.
2.3 Architecture
A GFS cluster consists of a single master and multiple
chunkservers and is accessed by multiple clients,asshown
in Figure 1. Each of these is typically a commodity Linux
machine running a user-level server process. It is easy to run
both a chunkserver and a client on the same machine, as long
as machine resources permit and the lower reliability caused
by running possibly flaky application code is acceptable.
Files are divided into fixed-size chunks. Each chunk is
identified by an immutable and globally unique 64 bit chunk
handle assigned by the master at the time of chunk creation.
Chunkservers store chunks on local disks as Linux files and
read or write chunk data specified by a chunk handle and
byte range. For reliability, each chunk is replicated on multi-
ple chunkservers. By default, we store three replicas, though
users can designate different replication levels for different
regions of the file namespace.
The master maintains all file system metadata. This in-
cludes the namespace, access control information, the map-
ping from files to chunks, and the current locations of chunks.
It also controls system-wide activities such as chunk lease
management, garbage collection of orphaned chunks, and
chunk migration between chunkservers. The master peri-
odically communicates with each chunkserver in HeartBeat
messages to give it instructions and collect its state.
GFS client code linked into each application implements
the file system API and communicates with the master and
chunkservers to read or write data on behalf of the applica-
tion. Clients interact with the master for metadata opera-
tions, but all data-bearing communication goes directly to
the chunkservers. We do not provide the POSIX API and
therefore need not hook into the Linux vnode layer.
Neither the client nor the chunkserver caches file data.
Client caches offer little benefit because most applications
stream through huge files or have working sets too large
to be cached. Not having them simplifies the client and
the overall system by eliminating cache coherence issues.
(Clients do cache metadata, however.) Chunkservers need
not cache file data because chunks are stored as local files
and so Linux’s buffer cache already keeps frequently accessed
data in memory.
2.4 Single Master
Having a single master vastly simplifies our design and
enables the master to make sophisticated chunk placement
of 15
免费下载
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文档的来源（墨天轮），文档链接，文档作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
下载排行榜
文档被以下合辑收录
评论