16.Testing Raft-Replicated Database Systems.pdf
免费下载
Testing Raft-Replicated Database
Systems
Guohao Ding
1
, Weining Qian
1(
B
)
, Peng Cai
1
, Tianze Pang
2
, and Qiong Zhao
2
1
East China Normal University, Shanghai, China
guohaoding@stu.ecnu.edu.cn, {wnqian,pcai}@dase.ecnu.edu.cn
2
Bank of Communications, Shanghai, China
{pangtz,qiongzhao}@bankcomm.com
Abstract. The replication technique based on Raft protocol is essential
in modern distributed and highly-available database systems. Although
Raft is a protocol easy to understand and implement, testing a Raft-
replicated database system is still a challenging task due to multiple
sources of nondeterminism. Conventional testing techniques, such as
unit, integration and stress testing, are ineffective in preventing seri-
ous but subtle bugs from reaching production. This paper first intro-
duces evaluation metrics after the abstraction of general Raft-replicated
database systems. These metrics are defined from several aspects includ-
ing correctness, performance, and scalability. Then, we present test
dimensions for the design of test cases, which contain various fault types,
different workloads and system configurations. Finally, we describe test
results of Raft-replicated open source database system.
Keywords: Raft
· Distributed database · Test
1 Introduction
In recent years, the amount of data created by human activities goes far beyond
the storage and processing power of a single computer. In order to process mas-
sive amounts of data, distributed systems, especially distributed database sys-
tems, are becoming more and more popular. It serves millions of users in many
important applications. However, distributed database systems are notoriously
hard to design, implement and test because they introduce more variables into
a design than a single machine does, making the root cause of an application
problem much harder to discover.
Developers of distributed systems use many testing techniques, such as unit
testing, integration testing, stress testing, and fault injection. In spite of exten-
sive use of these testing methods, many bugs that arise from subtle combina-
tions of concurrency and failure events that are missed during testing and get
exposed only in production. According to the interview [4], many technical lead-
ers and senior managers in famous companies, such as Microsoft, Amazon and
Google, have the consensus that one of the most critical problems today is how
c
Springer Nature Switzerland AG 2019
C. Zheng and J. Zhan (Eds.): Bench 2018, LNCS 11459, pp. 131–144, 2019.
https://doi.org/10.1007/978-3-030-32813-9
_12
132 G. Ding et al.
to improve testing coverage so that bugs can be uncovered during testing and
not in production.
Distributed consensus is a fundamental problem in distributed system. As
we all know, Paxos [8,9] is an important algorithm for solving distributed con-
sensus problems, proposed by Lamport in 1998, but over the years, few people
can really understand Paxos algorithm. Therefore, in order to let more people
understand the distributed consensus algorithm and also provide a better foun-
dation for building practical systems, Ongaro and Ousterhout proposed Raft [12]
protocol, which can be used as an alternative to the Paxos protocol. According
to the official website of the Raft protocol, there are currently over 50 different
implementations of Raft listed on their website. For a distributed system using
the Raft protocol, the correctness of the Raft implementation is critical, and
we must have a rigorous test. The amount of code used for testing should be
higher than the implementation Raft itself, which can be used as a reference for
open source implementation selection. The correctness of the Raft algorithm is
theoretically proven (specific reference paper [12]). That is to say, it is correct as
long as the algorithm is implemented correctly. But there is no uniform standard
on how to test the correctness of such systems, what metrics are needed to be
tested and how to design test cases.
In this paper, we define some evaluation metrics for testing distributed
database system based on the Raft protocol and propose how to design test
cases from different test dimensions. This further provides a reference for dis-
tributed system developers and testers to verify the correctness of the Raft-based
distributed system.
To the best of our knowledge, we are the first to systematically propose
test metrics and test dimensions of distributed database system based on Raft
protocol, and design hundreds of test cases to test the Raft-replicated database
systems in the actual production environment.
To summarize, our contributions are as follows:
– Abstract a general distributed database system model based on Raft protocol.
– Define the evaluation metrics for testing such system based on the Raft pro-
tocol and explain why these metrics are important.
– Describe the test dimensions and design hundreds of test cases to test Raft-
replicated open source database system in the actual production environment.
The remaining of this paper is organized as follows. Section 2 introduces the
background of replicated state machine problem and Raft. Section 3 gives a Raft-
based system model abstraction. Section 4 defines the evaluation metrics for
testing. Section 5 describes the different test dimensions. Section 6 evaluates the
performance of the Raft we implemented in the system. Section 7 presents related
work.
2 Background
This section first introduces the replicated state machine problem and then gives
an overview of basic Raft [12] protocol.
of 14
免费下载
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文档的来源（墨天轮），文档链接，文档作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
下载排行榜
评论