接上回，我们开始对两副本丢失进行演练。
从表region的分布图可以看到，当宕掉tikv2135、tikv5138两台主机情况下，整个集群并不会受到影响，因为只有一个region副本分布在这两台机器之上，但这仅仅是当数据库的数据两较小情况，当数据量增大PD调度将会对region的分布进行调度。对于挂掉一个副本的情况，在此不进行模拟。采用同时宕掉Tikv1134和Tikv3136这两台机器，会出现region的两个副本丢失：
先检查宕机前测试表的状况:
MySQL[sbtest2]> select count(*) from t_user;
+----------+
|count(*) |
+----------+
| 3000000 |
+----------+
1row in set (6.98 sec)
同时宕掉Tikv3 136和Tikv 4137两台机器后测试表的情况：
MySQL[sbtest2]> select count(*) from t_user;
ERROR9005 (HY000): Region is unavailable
正常的SQL语句出现region不可用的报错。
检查宕机的两台机器对应的store_id：
[root@tidb1bin]# root/tidb-v4.0.0-linux-amd64/bin/pd-ctl -i -uhttp://172.16.134.133:2379
»store
…
{
"store": {
"id": 5,
"address": "172.16.134.136:20160",
"labels": [
{
"key": "host",
"value": "tikv3"
}
],
"version": "4.0.0-rc",
"status_address": "172.16.134.136:20180",
"git_hash": "f45d0c963df3ee4b1011caf5eb146cacd1fbbad8",
"start_timestamp": 1594632461,
"binary_path":"/data1/tidb-deploy/tikv-20160/bin/tikv-server",
"last_heartbeat": 1594700897622993541,
"state_name": "Disconnected"
},…
"{
"store": {
"id": 4,
"address": "172.16.134.134:20160",
"labels": [
{
"key": "host",
"value": "tikv1"
}
],
"version": "4.0.0-rc",
"status_address": "172.16.134.134:20180",
"git_hash": "f45d0c963df3ee4b1011caf5eb146cacd1fbbad8",
"start_timestamp": 1594632462,
"binary_path":"/data1/tidb-deploy/tikv-20160/bin/tikv-server",
"last_heartbeat": 1594700897744383603,
"state_name": "Disconnected"
},
可以发现storeID 4和5状态名为“Disconnected”，一段时间后状态会成为“DOWN”。
通过 pd-ctlconfig get 获取region-schedule-limit、replica-schedule-limit、leader-schedule-limit、merge-schedule-limit
[root@tidb1bin]# ./pd-ctl -i -u http://172.16.134.133:2379
»config show
{
"replication": {
"enable-placement-rules": "false",
"location-labels": "host",
"max-replicas": 3,
"strictly-match-label": "false"
},
"schedule": {
"enable-cross-table-merge": "false",
"enable-debug-metrics": "false",
"enable-location-replacement": "true",
"enable-make-up-replica": "true",
"enable-one-way-merge": "false",
"enable-remove-down-replica": "true",
"enable-remove-extra-replica": "true",
"enable-replace-offline-replica": "true",
"high-space-ratio": 0.7,
"hot-region-cache-hits-threshold": 3,
"hot-region-schedule-limit": 4,
"leader-schedule-limit": 4,
"leader-schedule-policy": "count",
"low-space-ratio": 0.8,
"max-merge-region-keys": 200000,
"max-merge-region-size": 20,
"max-pending-peer-count": 16,
"max-snapshot-count": 3,
"max-store-down-time": "30m0s",
"merge-schedule-limit": 8,
"patrol-region-interval": "100ms",
"region-schedule-limit": 2048,
"replica-schedule-limit": 64,
"scheduler-max-waiting-operator": 5,
"split-merge-interval": "1h0m0s",
"store-balance-rate": 15,
"store-limit-mode": "manual",
"tolerant-size-ratio": 0
}
}
通过 pd-ctlconfig set 将这 4个参数设为 0
»config set region-schedule-limit 0
Success!
»config set replica-schedule-limit 0
Success!
»config set leader-schedule-limit 0
Success!
»config set merge-schedule-limit 0
Success!
关闭调度主要为将恢复过程中可能的异常情况降到最少，需在故障处理期间禁用相关的调度。
使用pd-ctl 检查大于等于一半副本数在故障节点上的Region，并记录它们的ID（故障节点为storeid 4，5）：
»region --jq=".regions[] | {id: .id, peer_stores:[.peers[].store_id] | select(length as $total | map(if .==(4,5) then. else empty end) | length>=$total-length) }"
{"id":3080,"peer_stores":[4,6,5]}
{"id":18,"peer_stores":[4,5,6]}
{"id":3084,"peer_stores":[4,6,5]}
{"id":75,"peer_stores":[4,5,6]}
{"id":34,"peer_stores":[6,4,5]}
{"id":4005,"peer_stores":[4,6,5]}
{"id":4009,"peer_stores":[5,6,4]}
{"id":83,"peer_stores":[4,5,6]}
{"id":3076,"peer_stores":[4,5,6]}
{"id":4013,"peer_stores":[5,4,6]}
{"id":10,"peer_stores":[4,6,5]}
{"id":26,"peer_stores":[4,6,5]}
{"id":59,"peer_stores":[4,5,6]}
{"id":3093,"peer_stores":[4,5,6]}
我们可以看到表的两个regionID均在列表中，另外的两个region由于只丢失一个副本，并未出现在列表中。
在剩余正常的kv节点上执行停Tikv的操作：
[root@tidb1bin]# tiup cluster stop tidb-test -R=tikv
Startingcomponent `cluster`: root/.tiup/components/cluster/v0.6.1/clusterstop tidb-test -R=tikv
+[ Serial ] - SSHKeySet:privateKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa,publicKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa.pub
+[Parallel] - UserSSH: user=tidb, host=172.16.134.133
+[Parallel] - UserSSH: user=tidb, host=172.16.134.133
+[Parallel] - UserSSH: user=tidb, host=172.16.134.134
+[Parallel] - UserSSH: user=tidb, host=172.16.134.135
+[Parallel] - UserSSH: user=tidb, host=172.16.134.136
+[Parallel] - UserSSH: user=tidb, host=172.16.134.137
+[Parallel] - UserSSH: user=tidb, host=172.16.134.138
+[Parallel] - UserSSH: user=tidb, host=172.16.134.133
+[Parallel] - UserSSH: user=tidb, host=172.16.134.133
+[Parallel] - UserSSH: user=tidb, host=172.16.134.133
+[ Serial ] - ClusterOperate: operation=StopOperation,options={Roles:[tikv] Nodes:[] Force:false SSHTimeout:5 OptTimeout:60APITimeout:300}
Stoppingcomponent tikv
Stopping instance 172.16.134.138
Stopping instance 172.16.134.134
Stopping instance 172.16.134.135
Stopping instance 172.16.134.136
Stopping instance 172.16.134.137
Stop tikv 172.16.134.135:20160 success
Stop tikv 172.16.134.138:20160 success
Stop tikv 172.16.134.137:20160 success
在所有健康的节点上执行(操作需要确保健康的节点关闭了Tikv)：
[root@tidb3bin]# ./tikv-ctl --db data1/tidb-data/tikv-20160/db unsafe-recoverremove-fail-stores -s 4,5 --all-regions
removingstores [4, 5] from configurations...
success
[root@tidb5bin]# ./tikv-ctl --db data1/tidb-data/tikv-20160/db unsafe-recoverremove-fail-stores -s 4,5 --all-regions
removingstores [4, 5] from configurations...
success
[root@tidb6bin]# ./tikv-ctl --db data1/tidb-data/tikv-20160/db unsafe-recoverremove-fail-stores -s 4,5 --all-regions
removingstores [4, 5] from configurations...
success
当然Region比较少，则可以在给定Region 的剩余副本上，移除掉所有位于故障节点上的Peer，在这些Region 的未发生掉电故障的机器上运行：
tikv-ctl--db path/to/tikv-data/db unsafe-recover remove-fail-stores -s<s1,s2> -r <r1,r2,r3>，对于region较多的情况，此操作则较为繁琐。
停止PD节点：
[root@tidb1~]# tiup cluster stop tidb-test -R=pd
Startingcomponent `cluster`: root/.tiup/component
重启启动PDtikv节点：
[root@tidb1~]# tiup cluster start tidb-test -R=pd,tikv
这里需要启动PD才能连接到数据库。
检查没有处于leader状态的region（要保持没有）：
[root@tidb1~]# pd-ctl -i -u http://172.16.134.133:2379
»region --jq '.regions[]|select(has("leader")|not)|{id:.id,peer_stores: [.peers[].store_id]}'
»
这里没有发现没有leader状态的region。
重新修改参数：
[root@tidb1~]# pd-ctl -i -u http://172.16.134.133:2379
»config set region-schedule-limit 2048
Success!
»config set replica-schedule-limit 64
Success!
»config set leader-schedule-limit 4
Success!
»config set merge-schedule-limit 8
Success!
检查查询数据是否正常
MySQL[sbtest2]> select count(*) from t_user;
+----------+
|count(*) |
+----------+
| 3000000 |
+----------+
1row in set (9.95 sec)
至此恢复操作结束。
我们再看看region的分布：
Region的副本进行了新的复制和分布。
完成了两副本的丢失的演练，三副本丢失会出现什么情况又该如何恢复？咱下回见。
参考文档https://book.tidb.io/session3/chapter5/recover-quorum.html
