Impala-A Modern, Open-Source SQL Engine for Hadoop.pdf
免费下载
Impala: A Modern, Open-Source SQL Engine for Hadoop
Marcel Kornacker Alexander Behm Victor Bittorf Taras Bobrovytsky
Casey Ching Alan Choi Justin Erickson Martin Grund Daniel Hecht
Matthew Jacobs Ishaan Joshi Lenni Kuff Dileep Kumar Alex Leblang
Nong Li Ippokratis Pandis Henry Robinson David Rorke Silvius Rus
John Russell Dimitris Tsirogiannis Skye Wanderman-Milne Michael Yoder
Cloudera
http://impala.io/
ABSTRACT
Cloudera Impala is a modern, open-source MPP SQL en-
gine architected from the ground up for the Hadoop data
processing environment. Impala provides low latency and
high concurrency for BI/analytic read-mostly queries on
Hadoop, not delivered by batch frameworks such as Apache
Hive. This paper presents Impala from a user’s perspective,
gives an overview of its architecture and main components
and briefly demonstrates its superior performance compared
against other popular SQL-on-Hadoop systems.
1. INTRODUCTION
Impala is an open-source
1
, fully-integrated, state-of-the-
art MPP SQL query engine designed specifically to leverage
the flexibility and scalability of Hadoop. Impala’s goal is
to combine the familiar SQL support and multi-user perfor-
mance of a traditional analytic database with the scalability
and flexibility of Apache Hadoop and the production-grade
security and management extensions of Cloudera Enterprise.
Impala’s beta release was in October 2012 and it GA’ed in
May 2013. The most recent version, Impala 2.0, was released
in October 2014. Impala’s ecosystem momentum continues
to accelerate, with nearly one million downloads since its
GA.
Unlike other systems (often forks of Postgres), Impala is a
brand-new engine, written from the ground up in C++ and
Java. It maintains Hadoop’s flexibility by utilizing standard
components (HDFS, HBase, Metastore, YARN, Sentry) and
is able to read the majority of the widely-used file formats
(e.g. Parquet, Avro, RCFile). To reduce latency, such as
that incurred from utilizing MapReduce or by reading data
remotely, Impala implements a distributed architecture based
on daemon processes that are responsible for all aspects of
query execution and that run on the same machines as the
rest of the Hadoop infrastructure. The result is performance
1
https://github.com/cloudera/impala
This article is published under a Creative Commons Attribution Li-
cense(http://creativecommons.org/licenses/by/3.0/), which permits distri-
bution and reproduction in any medium as well as allowing derivative
works, provided that you attribute the original work to the author(s) and
CIDR 2015.
7th Biennial Conference on Innovative Data Systems Research (CIDR’15)
January 4-7, 2015, Asilomar, California, USA.
that is on par or exceeds that of commercial MPP analytic
DBMSs, depending on the particular workload.
This paper discusses the services Impala provides to the
user and then presents an overview of its architecture and
main components. The highest performance that is achiev-
able today requires using HDFS as the underlying storage
manager, and therefore that is the focus on this paper; when
there are notable differences in terms of how certain technical
aspects are handled in conjunction with HBase, we note that
in the text without going into detail.
Impala is the highest performing SQL-on-Hadoop system,
especially under multi-user workloads. As
Section 7
shows,
for single-user queries, Impala is up to 13x faster than alter-
natives, and 6.7x faster on average. For multi-user queries,
the gap widens: Impala is up to 27.4x faster than alternatives,
and 18x faster on average – or nearly three times faster on
average for multi-user queries than for single-user ones.
The remainder of this paper is structured as follows: the
next section gives an overview of Impala from the user’s
perspective and points out how it differs from a traditional
RDBMS.
Section 3
presents the overall architecture of the
system.
Section 4
presents the frontend component, which
includes a cost-based distributed query optimizer,
Section 5
presents the backend component, which is responsible for the
query execution and employs runtime code generation, and
Section 6
presents the resource/workload management com-
ponent.
Section 7
briefly evaluates the performance of Im-
pala.
Section 8
discusses the roadmap ahead and
Section 9
concludes.
2. USER VIEW OF IMPALA
Impala is a query engine which is integrated into the
Hadoop environment and utilizes a number of standard
Hadoop components (Metastore, HDFS, HBase, YARN, Sen-
try) in order to deliver an RDBMS-like experience. However,
there are some important differences that will be brought up
in the remainder of this section.
Impala was specifically targeted for integration with stan-
dard business intelligence environments, and to that end
supports most relevant industry standards: clients can con-
nect via ODBC or JDBC; authentication is accomplished
with Kerberos or LDAP; authorization follows the standard
SQL roles and privileges
2
. In order to query HDFS-resident
2
This is provided by another standard Hadoop component
called Sentry
[4]
, which also makes role-based authoriza-
tion available to Hive, and other components.
data, the user creates tables via the familiar
CREATE TABLE
statement, which, in addition to providing the logical schema
of the data, also indicates the physical layout, such as file
format(s) and placement within the HDFS directory struc-
ture. Those tables can then be queried with standard SQL
syntax.
2.1 Physical schema design
When creating a table, the user can also specify a list of
partition columns:
CREATE TABLE T (...) PARTITIONED BY (day int, month
int) LOCATION ’<hdfs-path>’ STORED AS PARQUET;
For an unpartitioned table, data files are stored by de-
fault directly in the root directory
3
. For a partitioned
table, data files are placed in subdirectories whose paths
reflect the partition columns’ values. For example, for day
17, month 2 of table T, all data files would be located in
directory
<root>/day=17/month=2/
. Note that this form of
partitioning does not imply a collocation of the data of an
individual partition: the blocks of the data files of a partition
are distributed randomly across HDFS data nodes.
Impala also gives the user a great deal of flexibility when
choosing file formats. It currently supports compressed and
uncompressed text files, sequence file (a splittable form of
text files), RCFile (a legacy columnar format), Avro (a binary
row format), and Parquet, the highest-performance storage
option (
Section 5.3
discusses file formats in more detail).
As in the example above, the user indicates the storage
format in the
CREATE TABLE
or
ALTER TABLE
statements. It
is also possible to select a separate format for each partition
individually. For example one can specifically set the file
format of a particular partition to Parquet with:
ALTER TABLE PARTITION(day=17, month=2) SET FILEFORMAT
PARQUET.
As an example for when this is useful, consider a table
with chronologically recorded data, such as click logs. The
data for the current day might come in as CSV files and get
converted in bulk to Parquet at the end of each day.
2.2 SQL Support
Impala supports most of the SQL-92
SELECT
statement
syntax, plus additional SQL-2003 analytic functions, and
most of the standard scalar data types: integer and floating
point types, STRING, CHAR, VARCHAR, TIMESTAMP,
and DECIMAL with up to 38 digits of precision. Custom
application logic can be incorporated through user-defined
functions (UDFs) in Java and C++, and user-defined aggre-
gate functions (UDAs), currently only in C++.
Due to the limitations of HDFS as a storage manager, Im-
pala does not support
UPDATE
or
DELETE
, and essentially only
supports bulk insertions (
INSERT INTO ... SELECT ...
)
4
.
Unlike in a traditional RDBMS, the user can add data to a
table simply by copying/moving data files into the directory
3
However, all data files that are located in any directory
below the root are part of the table’s data set. That is a
common approach for dealing with unpartitioned tables,
employed also by Apache Hive.
4
We should also note that Impala supports the
VALUES
clause. However, for HDFS-backed tables this will generate
one file per
INSERT
statement, which leads to very poor
performance for most applications. For HBase-backed
tables, the
VALUES
variant performs single-row inserts by
means of the HBase API.
location of that table, using HDFS’s API. Alternatively, the
same can be accomplished with the LOAD DATA statement.
Similarly to bulk insert, Impala supports bulk data dele-
tion by dropping a table partition (
ALTER TABLE DROP PAR-
TITION
). Because it is not possible to update HDFS files
in-place, Impala does not support an
UPDATE
statement. In-
stead, the user typically recomputes parts of the data set to
incorporate updates, and then replaces the corresponding
data files, often by dropping and re-adding the partition
After the initial data load, or whenever a significant frac-
tion of the table’s data changes, the user should run the
COMPUTE STATS <table>
statement, which instructs Impala
to gather statistics on the table. Those statistics will subse-
quently be used during query optimization.
3. ARCHITECTURE
Impala is a massively-parallel query execution engine,
which runs on hundreds of machines in existing Hadoop
clusters. It is decoupled from the underlying storage engine,
unlike traditional relational database management systems
where the query processing and the underlying storage engine
are components of a single tightly-coupled system. Impala’s
high-level architecture is shown in Figure 1.
An Impala deployment is comprised of three services. The
Impala daemon (impalad) service is dually responsible for
accepting queries from client processes and orchestrating their
execution across the cluster, and for executing individual
query fragments on behalf of other Impala daemons. When
an Impala daemon operates in the first role by managing
query execution, it is said to be the coordinator for that query.
However, all Impala daemons are symmetric; they may all
operate in all roles. This property helps with fault-tolerance,
and with load-balancing.
One Impala daemon is deployed on every machine in the
cluster that is also running a datanode process - the block
server for the underlying HDFS deployment - and therefore
there is typically one Impala daemon on every machine. This
allows Impala to take advantage of data locality, and to read
blocks from the filesystem without having to use the network.
The Statestore daemon (statestored) is Impala’s meta-
data publish-subscribe service, which disseminates cluster-
wide metadata to all Impala processes. There is a single
statestored instance, which is described in more detail in
Section 3.1 below.
Finally, the Catalog daemon (catalogd), described in
Section 3.2
,
serves as Impala’s catalog repository and metadata access
gateway. Through the catalogd, Impala daemons may exe-
cute DDL commands that are reflected in external catalog
stores such as the Hive Metastore. Changes to the system
catalog are broadcast via the statestore.
All these Impala services, as well as several configuration
options, such as the sizes of the resource pools, the available
memory, etc.. (see
Section 6
for more details about resource
and workload management) are also exposed to Cloudera
Manager, a sophisticated cluster management application
5
.
Cloudera Manager can administer not only Impala but also
pretty much every service for a holistic view of a Hadoop
deployment.
5
http://www.cloudera.com/content/cloudera/en/products-
and-services/cloudera-enterprise/cloudera-manager.html
of 10
免费下载
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文档的来源（墨天轮），文档链接，文档作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
下载排行榜
评论