SparkCruise- Workload Optimization in Managed Spark Clusters at Microsof.pdf

Dylan

191

13页

0次

2022-07-11

5墨值下载

SparkCruise: Workload Optimization in Managed Spark

Clusters at Microso

Abhishek Roy

Microsoft

abhishek.roy@microsoft.com

Alekh Jindal

Microsoft

alekh.jindal@microsoft.com

Priyanka Gomatam

Microsoft

priyanka.gomatam@microsoft.com

Xiating Ouyang

∗

University of Wisconsin-Madison

xouyang@cs.wisc.edu

Ashit Gosalia

Microsoft

ashit.gosalia@microsoft.com

Nishkam Ravi

Microsoft

nishkam.ravi@microsoft.com

Swinky Mann

Microsoft

swinky.mann@microsoft.com

Prakhar Jain

∗

Databricks

prakhar.jain@databricks.com

ABSTRACT

Today cloud companies oer fully managed Spark services. This

has made it easy to onboard new customers but has also increased

the volume of users and their workload sizes. However, both cloud

providers and users lack the tools and time to optimize these mas-

sive workloads. To solve this problem, we designed SparkCruise that

can help understand and optimize workload instances by adding a

workload-driven feedback loop to the Spark query optimizer. In this

paper, we present our approach to collecting and representing Spark

query workloads and use it to improve the overall performance on

the workload, all without requiring any access to user data. These

methods scale with the number of workloads and apply learned

feedback in an online fashion. We explain one specic workload

optimization developed for computation reuse. We also share the

detailed analysis of production Spark workloads and contrast them

with the corresponding analysis of TPC-DS benchmark. To the best

of our knowledge, this is the rst study to share the analysis of

large-scale production Spark SQL workloads.

PVLDB Reference Format:

Abhishek Roy, Alekh Jindal, Priyanka Gomatam, Xiating Ouyang, Ashit

Gosalia, Nishkam Ravi, Swinky Mann, and Prakhar Jain. SparkCruise:

Workload Optimization in Managed Spark Clusters at Microsoft. PVLDB,

14(12): 3122 - 3134, 2021.

doi:10.14778/3476311.3476388

1 INTRODUCTION

Spark [

] is a widely popular data processing platform that is used

for a variety of analytical tasks, including batch processing, interac-

tive exploration, streaming analytics, graph analytics, and machine

learning. At Microsoft, Azure HDInsight [

] oers managed Spark

clusters that allow users to start processing their data processing

without worrying about managing the underlying infrastructure.

This work is licensed under the Creative Commons BY-NC-ND 4.0 International

License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of

this license. For any use beyond those covered by this license, obtain permission by

emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights

licensed to the VLDB Endowment.

Proceedings of the VLDB Endowment, Vol. 14, No. 12 ISSN 2150-8097.

doi:10.14778/3476311.3476388

However, once the data processing tasks are deployed as produc-

tion workows, users are largely responsible for optimizing their

workloads to achieve better performance at lower costs. This is fast

emerging as a major pain in cloud data services, more so due to

the lack of DBAs in the cloud environments [

], leading to several

recent eorts for building new platforms and features that optimize

end to end workloads in the cloud [22, 27, 29, 30].

Performance improvements in Spark have come a long way over

the last decade. It started with the initial proposal of resilient dis-

tributed datasets (RDDs) in 2010 for improving the performance of

multiple parallel operations by reusing a working set of data [

Later, Shark [

] was proposed in 2013 to run declarative Hive

queries (SQL-on-Hadoop) interactively using the Spark processing

backend. The Shark project evolved into Spark SQL [

] for do-

ing relational data processing along with a query optimizer, called

Catalyst, in 2015. While Catalyst was rule-based in the beginning,

query costing and cost-based query optimization was later added

to Spark in 2017 [

]. Most recently, given the broader realization

that it is often hard to make the right query optimization choices

at compile time [

], adaptive query execution was introduced in

Spark in 2020 [

]. However, given the breadth of applications and

deployment scenarios that are typically seen in modern data pro-

cessing systems like Spark, it is still hard to pre-build the right set

of optimizations in the system itself. This has led to a new wave of

thinking to instance optimize a data processing system to a given

workload [

]. In fact, the presence of hundreds of congu-

rations in current Spark codebase aligns with the above line of

thinking that a Spark deployment could be tuned to dierent work-

load needs. Unfortunately, it is incredibly hard, if not impossible, to

manually tune these congs or adapt the system to a given work-

load. Interestingly, modern cloud deployments of data processing

systems oer an unprecedented opportunity to observe and learn

from large volumes of workloads. As a result, we could build a

workload-driven feedback loop to automatically (and continuously)

tune the system from the workloads seen at hand.

We presented our overarching vision on how to improve cloud

query engines in [

]. In this paper, we describe SparkCruise, the

next big step in optimizing Spark workloads that we have built for

∗

Work done while at Microsoft.

3122

Spark clusters in Azure HDInsight. SparkCruise exposes a work-

load optimization platform that leverages massive cloud workloads

and provides a feedback loop to the Spark engine for improving

performance and reducing costs. We demonstrated an early ver-

sion of the SparkCruise system earlier [

]. Since then we have

added newer techniques for plan log collection, introduced a scal-

able telemetry pipeline that runs daily, explored data cleaning and

integration techniques to improve the quality of our common work-

load representation, analyzed production workloads to characterize

the Spark workloads in HDInsight, provided a notebook for cus-

tomers to derive insights from their own workloads, and pushed

one concrete feature for automatic computation reuse all the way to

production. We describe the overall system design and extensibility

of SparkCruise, the opportunities for compute reuse in production

Spark workloads, the deployment of SparkCruise in HDInsight,

and the experiences from our production journey.

Our key contributions can be summarized as follows:

•

We present the SparkCruise platform for adding workload-

driven feedback loop in Spark, and discuss how it transforms

Spark engine from optimizing one query at a time to opti-

mizing end to end workloads. (Section 3)

•

We describe a query plan telemetry pipeline for collecting

anonymized Spark SQL plans with low overheads and at

production scale. (Section 4)

•

We introduce a denormalized workload representation for

Spark that combines both the compile-time and run-time

characteristics of the workload and could be used for a va-

riety of optimization features. We discuss the data quality

challenges in creating this workload representation and show

cleaning techniques to overcome them. (Section 5)

•

We present detailed insights from production Spark work-

loads at Microsoft, including distributions of inputs, applica-

tions, queries, operators, cardinalities, selectivities, and plan

shapes such as width and height. (Section 6)

•

We describe a workload insights notebook that we have built

and released for customers in HDInsight to discover insights

from their own workloads. (Section 7)

•

Finally, we drill down into automatic computation reuse as a

concrete workload optimization in Spark that we have built

and released for customers in HDInsight. We discuss the

reuse mechanisms and various online and oine policies for

view selection and materialization. (Section 8)

2 SPARK BACKGROUND

The Spark data processing platform supports a variety of analytical

applications including batch or interactive analytics over structured

or unstructured data, streaming analytics over constantly arriving

data, graph analytics over linked data, iterative machine learning

algorithms, and the newer data science applications. Structured

data processing, in particular, has increasingly gained enterprise

level adoption in the last few years with several large companies

running their key ETL workloads using Spark. This has resulted

in several trends. First, Spark has become the most active Apache

project that is visited on GitHub [

], with a vibrant open source

community of 83 committers [

] and numerous meet-ups around

the world [

], Second, there is in-house Spark development at

several large enterprises such as LinkedIn [

], Facebook [

], and

IBM [

], and Third, there are managed Spark services from all major

cloud providers, including Amazon Web Services [

], Microsoft

Azure [6, 7, 26], and Google Cloud [10].

At Microsoft, Azure HDInsight allows customers to run popular

open source frameworks — including Apache Hadoop, Spark, Hive,

Kafka, and more [

]. Essentially, it abstracts the complexities in

setting up and maintaining the cluster, and providing a more man-

aged experience for customers to quickly get started with their

analytical tasks. For Spark, this means that users can leverage the

latest Spark distributions, easily congure their cluster for dierent

application needs, and monitor and tune the performance and costs.

As a result of this better Spark infrastructure experience, we nd

a large fraction of HDInsight customers running their recurring

ETL workloads. Others prominent use of HDInsight Spark is for

interactive notebooks that have become very popular for ad-hoc

analysis. Interestingly, workload optimization is relevant to both

these usage types: for saving total costs in ETL workloads and for

reducing the time to insights in interactive workloads.

Spark workloads are made up of applications, each of which

consist of one or more queries running in the Spark session. Multiple

applications can run in parallel on the same cluster. We focus on

Spark SQL queries, i.e, all analytics that compile down to Spark

dataframes and go through the Catalyst query optimizer, while

ignoring the programs written directly against the RDDs. This is

because declarative Spark SQL workloads are more amenable to

characterization and feedback in the query optimizer layer (without

aecting the user expectation on how the programs should be

executed, as with RDDs), not to mention they also form the majority

of our workloads.

In the remainder of the paper, we rst provide an overview of

SparkCruise, our workload optimization platform for Spark, before

describing each of its components and discussing the features we

have shipped in HDInsight.

3 SPARKCRUISE OVERVIEW

SparkCruise adds a workload-driven feedback loop to Spark to

instance optimize its performance for a given workload. Figure 1

shows the overall architecture. As mentioned before, we focus on

Spark SQL queries that run through the Catalyst query optimizer

and that users expect the system to optimize, as opposed to RDD

programs that are almost like physical execution plans handcrafted

by the users. There are four sets of components in Figure 1 that are

worth highlighting and we discuss them below.

First, SparkCruise provides an elaborate query plan telemetry

that captures Spark SQL query plans in a scalable manner. This

includes an additional plan log listener to collect plans in JSON

format, adding identiers called signatures at each node in the

query plan, anonymizing the plans from any personally identiable

information (PII), and collecting the resulting log in both structured

and semi-structured format with varying degree of retention. The

query plan telemetry is enabled simply via a conguration change

and once collected it could be used for a variety of further analysis

by both the service provider and well as the customer themselves.

Second, the workload collected above goes through a set of pre-

processing to generate a common workload representation that

3123

of 13

5墨值下载

spark 优化

关注

评论