HBase 实践 | AWS EMR HBase 超大表迁移、备份、还原、同步演练手册：全量快照 + 实时同步不停机迁移方案

1. 背景介绍

2. 知识储备

2.1 HBase Snapshot

2.2 HBase Replication

1. 数据同步是 RegionServer 和 RegionServer 之间对拷的 （所以 Replication 有一个限速配置：set_peer_bandwidth，限制的也是单台 RegionServer 用于同步 WAL 日志的带宽）
2. RegionServer 会启动独立的线程异步执行 WAL 日志推送和接收
3. 为了保证Region Server 和 Region Server 之间对拷数据的有序性，WAL Log 的 edits 是以队列形式按顺序推送到目标端并进行回放的
4. 在目标端 edits 是以批量方式写入目标表的

3. 方案介绍

1. 在 Source Cluster 上将 Sink Cluster 添加为 peer，但需先 disable peer，然后再 enable 目标表的 replication，目的是让 Source Cluster 在创建前快照前先积累 WAL 日志，避免迁移过程中丢失数据
2. 在 Source Cluster 上创建快照
3. 在 Source Cluster 上将创建好的快照导出到 S3
4. 在 Sink Cluster 上还原快照
5. 在 Source Cluster 上 enable peer，恢复数据同步，直至数据追平

4. 环境说明

5. 演练操作

• 本演练的多数命令都需执行较长时间，因此，命令都使用了 nohup ... &
   形式转为了后台运行，以防 Linux 终端 Session 超时导致命令意外终止；
• 本演练的操作必须按顺序执行，不可在前一个命令转为后台运行后，立即执行下一条命令；
• 演练过程中需要在 Source Cluster、Sink Cluster 和 AWS Console 上来回切换，为了提醒操作者，本文特意使用 [ 执行环境 ] :: ...
   形式的标题强调本节操作的执行环境

5.1 [ AWS Console ] :: 创建 Source Cluster

[
  {
    "Classification": "hdfs-site",
    "Properties": {
      "dfs.replication": "1"
    }
  },
  {
    "Classification": "core-site",
    "Properties": {
      "fs.s3.awsAccessKeyId": "<your_aws_access_key_id>",
      "fs.s3.awsSecretAccessKey": "<your_aws_secret_access_key>"
    }
  },
  {
    "Classification": "yarn-site",
    "Properties": {
      "yarn.nodemanager.resource.memory-mb": "61440",
      "yarn.scheduler.maximum-allocation-mb": "61440"
    }
  },
  {
    "Classification": "hbase-env",
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
          "HBASE_MASTER_OPTS": "\"$HBASE_MASTER_OPTS -Xmx60g\"",
          "HBASE_REGIONSERVER_OPTS": "\"$HBASE_REGIONSERVER_OPTS -Xmx60g\""
        }
      }
    ],
    "Properties": {}
  },
  {
    "Classification": "hbase-site",
    "Properties": {
      "hbase.rpc.timeout": "86400000",
      "hbase.client.operation.timeout": "86400000",
      "hbase.client.scanner.timeout.period": "86400000",
      "hbase.client.sync.wait.timeout.msec": "86400000",
      "hbase.client.procedure.future.get.timeout.msec": "86400000"
    }
  }
]

5.2 [ Source Cluster ] :: 全局变量

export SINK_CLUSTER_HBASE_ROOT_DIR="s3://<your-hbase-sink-cluster-location>"
export TABLE_NAME="usertable"
export SNAPSHOT_NAME="usertable-snapshot"
export YCSB_VERSION="0.17.0"
export HBASE_VERSION="hbase14"
export YCSB_HOME="/opt/ycsb-${HBASE_VERSION}-binding-${YCSB_VERSION}"
export YCSB_HISTORICAL_RECORD_COUNT=5242880 # history data volume: 50 TB
export YCSB_INCREMENTAL_RECORD_COUNT=524288 # incremental data volume: 5 TB
export YCSB_INCREMENTAL_RECORD_BATCH=0
export WORKER_NODES=25
export REGIN_SPLITS=$((9780-1))
export SNAPSHOT_EXPORT_MAPPERS=198
export YCSB_THREADS=$WORKER_NODES

5.3 [ Source Cluster ] :: 创建目标桶

aws s3 mb $SINK_CLUSTER_HBASE_ROOT_DIR
# clean up in case the bucket is already in used, especially when re-run 
nohup aws s3 rm --recursive $SINK_CLUSTER_HBASE_ROOT_DIR &> rm-root-dir.out &
tail -f rm-root-dir.out

5.4 [ Source Cluster ] :: 监控数据表目录容量

pkill -f du-hdfs-s3
cat << EOF > /tmp/du-hdfs-s3.sh
for i in {1..1008};do
    echo -ne "[ \$(date +'%F %R') ]\n\n"
    hdfs dfs -du -h /user/hbase/
    dataVolume=\$(aws s3 ls --summarize --human-readable --recursive $SINK_CLUSTER_HBASE_ROOT_DIR/data/default/ | \
    sed -n "s/\s*Total Size:\s\(.*\)/\1/p")
    echo "\$dataVolume  $SINK_CLUSTER_HBASE_ROOT_DIR/data/default/" 
    archiveVolume=\$(aws s3 ls --summarize --human-readable --recursive $SINK_CLUSTER_HBASE_ROOT_DIR/archive/data/default/ | \
    sed -n "s/\s*Total Size:\s\(.*\)/\1/p")
    echo "\$archiveVolume  $SINK_CLUSTER_HBASE_ROOT_DIR/archive/data/default/" 
    sleep 600
done
EOF
nohup sh /tmp/du-hdfs-s3.sh &> du-hdfs-s3.out &
tail -f du-hdfs-s3.out

5.5 [ Source Cluster ] :: 创建测试表

cat << EOF | sudo -u hbase hbase shell
create '${TABLE_NAME}', 'cf', {SPLITS => (1..${REGIN_SPLITS}).map {|i| "user#{10000+i*(99999-10000)/${REGIN_SPLITS}}"}}
describe '${TABLE_NAME}'
EOF

5.6 [ Source Cluster ] :: 安装 YCSB

YCSB_URL="https://github.com/brianfrankcooper/YCSB/releases/download/${YCSB_VERSION}/ycsb-${HBASE_VERSION}-binding-${YCSB_VERSION}.tar.gz"
wget -c $YCSB_URL -P /tmp/
sudo tar -xzf /tmp/$(basename $YCSB_URL) -C /opt

5.7 [ Source Cluster ] :: 压入全量数据

sudo pkill -f ycsb
nohup sudo -u hbase $YCSB_HOME/bin/ycsb load $HBASE_VERSION \
    -cp /etc/hbase/conf/ \
    -p table=$TABLE_NAME \
    -p columnfamily=cf \
    -p recordcount=$YCSB_HISTORICAL_RECORD_COUNT \
    -p fieldcount=10 \
    -p fieldlength=1048576 \
    -p workload=site.ycsb.workloads.CoreWorkload \
    -p clientSideBuffering=true \
    -p writebuffersize=34359738368 \
    -threads $YCSB_THREADS \
    -s &> ycsb-historical-load.out &

tail -f ycsb-historical-load.out

5.8 ※ 等待 25 小时 ※

• 用于存储表数据的文件夹 /user/hbase/data/default/usertable
   其数据量整体保持持续上涨态势，但在固定周期内会有一定的小幅缩减，这是因为在周期内发生了 Compaction，文件总体积会有一定的缩减；
• 用于存储表归档文件的文件夹 /user/hbase/archive/data/default/usertable
   其数据量维持在几百 GB 到 1TB 之间波动，原因是：周期性执行 Compaction 后，旧的 HFile 会被移动到该文件夹下，此时文件夹的总容量是上涨的，但是另一个周期性的清理线程又会定期删除该文件夹下的数据，所以该文件夹的总容量不会一直增长，而是在一个范围内波动。

5.9 [ Source Cluster ] :: 检查全量数据

hdfs dfs -du -h /user/hbase/data/default/ /user/hbase/archive/data/default/

nohup sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.RowCounter $TABLE_NAME &> rowcounter-$TABLE_NAME.out &
tail -f rowcounter-$TABLE_NAME.out

nohup sudo -u hbase hbase shell <<< "count '$TABLE_NAME', INTERVAL => 1000000" &> count.out &
tail -f count.out

5.10 [ Source Cluster ] :: 压入增量数据

# incremental insert, retry every 60 seconds if write failed, keep trying 6 hours.
cat << EOF > incremental.workload
recordcount=$YCSB_INCREMENTAL_RECORD_COUNT
insertstart=$((YCSB_HISTORICAL_RECORD_COUNT + (YCSB_INCREMENTAL_RECORD_BATCH++) * YCSB_INCREMENTAL_RECORD_COUNT))
fieldcount=10
fieldlength=1048576
workload=site.ycsb.workloads.CoreWorkload
clientSideBuffering=true
writebuffersize=1073741824
core_workload_insertion_retry_limit=360
core_workload_insertion_retry_interval=60
EOF

# downgrade concurrency, 1 thread only.
nohup sudo -u hbase $YCSB_HOME/bin/ycsb load $HBASE_VERSION \
    -cp /etc/hbase/conf/ \
    -p table=$TABLE_NAME \
    -p columnfamily=cf \
    -P incremental.workload \
    -threads 1 \
    -s &> ycsb-incremental-load-$YCSB_INCREMENTAL_RECORD_BATCH.out &

tail -f ycsb-incremental-load-$YCSB_INCREMENTAL_RECORD_BATCH.out

5.11 [ AWS Console ] :: 创建 Sink Cluster

jq . << EOF | tee sink-cluster.json | jq
[
  {
    "Classification": "hdfs-site",
    "Properties": {
      "dfs.replication": "3"
    }
  },
  {
    "Classification": "yarn-site",
    "Properties": {
      "yarn.nodemanager.resource.memory-mb": "61440",
      "yarn.scheduler.maximum-allocation-mb": "61440"
    }
  },
  {
    "Classification": "hbase-env",
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
          "HBASE_MASTER_OPTS": "\"\$HBASE_MASTER_OPTS -Xmx60g\"",
          "HBASE_REGIONSERVER_OPTS": "\"\$HBASE_REGIONSERVER_OPTS -Xmx60g\""
        }
      }
    ],
    "Properties": {}
  },
  {
    "Classification": "hbase",
    "Properties": {
      "hbase.emr.storageMode": "s3"
    }
  },
  {
    "Classification": "emrfs-site",
    "Properties": {
      "fs.s3.maxConnections": "50000"
    }
  },
  {
    "Classification": "hbase-site",
    "Properties": {
      "hbase.rootdir": "$SINK_CLUSTER_HBASE_ROOT_DIR/",
      "hbase.hregion.majorcompaction": "0",
      "hbase.master.cleaner.interval": "604800000",
      "hbase.rpc.timeout": "86400000",
      "hbase.client.sync.wait.timeout.msec": "86400000",
      "hbase.client.operation.timeout": "86400000",
      "hbase.client.scanner.timeout.period": "86400000",
      "hbase.client.procedure.future.get.timeout.msec": "86400000",
      "hbase.snapshot.master.timeoutMillis": "86400000",
      "hbase.snapshot.master.timeout.millis": "86400000",
      "hbase.snapshot.region.timeout": "86400000"
    }
  }
]
EOF

sudo yum -y install xmlstarlet
xmlstarlet sel -t -v '/configuration/property[name = "hbase.zookeeper.quorum"]/value' -n /etc/hbase/conf/hbase-site.xml

# run on [ Source Cluster ] !
export SINK_CLUSTER_HBASE_MASTER_HOST="<your-sink-cluster-hbase-master-host-list>"

5.12 ※ 暂停写入 ※

5.13 [ Source Cluster ] :: 建立与 Sink Cluster 的同步

cat << EOF | nohup sudo -u hbase hbase shell &> enable-replication.out &
# disable table first...
disable '${TABLE_NAME}'
# replication config ops...
alter '${TABLE_NAME}',{NAME => 'cf', REPLICATION_SCOPE => '1'}
add_peer 'sink_cluster', CLUSTER_KEY => '$SINK_CLUSTER_HBASE_MASTER_HOST:2181:/hbase'
disable_peer 'sink_cluster'
list_peers
enable_table_replication '$TABLE_NAME'
list_replicated_tables
status 'replication'
# enable table
enable '${TABLE_NAME}'
EOF
tail -f enable-replication.out

5.14 [ Source Cluster ] :: 创建快照

# disble table
sudo -u hbase hbase shell <<< "disable '$TABLE_NAME'"
# create snapshot
sudo -u hbase hbase snapshot create -n $SNAPSHOT_NAME -t $TABLE_NAME
# sleep for a while, snapshot may be NOT available immediately.
sleep 120
# check if snapshot is available
sudo -u hbase hbase snapshot info -list-snapshots
# enable table
sudo -u hbase hbase shell <<< "enable '$TABLE_NAME'"

5.15 ※ 恢复写入 ※

5.16 [ Source Cluster ] :: 导出快照

# export snapshot to sink cluster hbase root dir
nohup sudo -u hbase hbase snapshot export \
    -snapshot $SNAPSHOT_NAME \
    -copy-to "$SINK_CLUSTER_HBASE_ROOT_DIR/" \
    -mappers $SNAPSHOT_EXPORT_MAPPERS &> export-snapshot.out &
tail -f /var/log/hbase/hbase.log

5.17 ※ 等待 3 小时 ※

5.18 [ Sink Cluster ] :: 还原快照

• 确认快照已存在

  sudo -u hbase hbase snapshot info -list-snapshots

• 还原快照

  cat << EOF | nohup sudo -u hbase hbase shell &> restore-snapshot.out &
  disable '$TABLE_NAME'
  restore_snapshot '$SNAPSHOT_NAME'
  enable '$TABLE_NAME'
  EOF
  tail -f restore-snapshot.out
  

5.19 ※ 等待 2 小时 ※

5.20 [ Sink Cluster ] :: 修改配置并重启

5.21 [ Source Cluster ] :: 放开与 Sink Cluster 的同步

pkill -f status-replication
cat << EOF > /tmp/status-replication.sh
for i in {1..1008};do
    sudo -u hbase hbase shell <<< "status 'replication'"
    sleep 60
done
EOF
nohup sh /tmp/status-replication.sh &> status-replication.out &
tail -f status-replication.out

sudo -u hbase hbase shell <<< "enable_peer 'sink_cluster'"

5.22 ※ 等待日志追平 ※

5.23 [ Sink Cluster ] :: 使用新配置创建新集群

1. 暂停 Source Cluster 的写入
2. 关停旧集群（具体操作请遵循此官方文档：[ HBase on Amazon S3 (Amazon S3 storage mode) ] 的 “Shutting down and restoring a cluster without data loss” 一节）
3. 移除 Source Cluster 与 旧的 Sink Cluster 之间的 replication 关系
4. 使用正常配置创建新的 Sink Cluster
5. 配置 Source Cluster 与新的 Sink Cluster 之间的 replication 关系
6. 恢复 Source Cluster 的写入

5.24 ※ 核对数据 ※

sudo pkill -f ycsb

nohup sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.RowCounter $TABLE_NAME &> rowcounter-$TABLE_NAME.out &
tail -f /var/log/hbase/hbase.log

6. 资源清理

6.1 移除同步

cat << EOF | sudo -u hbase hbase shell &> remove-peer.out &
disable_table_replication '$TABLE_NAME'
alter '${TABLE_NAME}',{NAME => 'cf',REPLICATION_SCOPE => '0'}
disable_peer 'sink_cluster'
remove_peer 'sink_cluster'
status 'replication'
EOF
tail -f remove-peer.out

6.2 删除快照

sudo -u hbase hbase shell <<< "delete_snapshot '$SNAPSHOT_NAME'"

6.3 删除测试表

cat << EOF | sudo -u hbase hbase shell &> drop-table.out &
truncate '${TABLE_NAME}'
disable '${TABLE_NAME}'
drop '${TABLE_NAME}'
EOF
tail -f drop-table.out

7. 已知错误

7.1 导出快照时报：Unable to load AWS credentials 错误

ERROR [main] snapshot.ExportSnapshot: Snapshot export failed
....
Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: 
....

7.2 还原快照时报：can’t find hfile 错误

7.3 HDFS 空间耗尽

7.4 HBase Shell 报 ERROR：The procedure x is still running 或 Snapshot … wasn’t completed in expectedTime 错误

@shell.hbase.configuration.setInt("hbase.rpc.timeout", 86400000)
@shell.hbase.configuration.setInt("hbase.client.operation.timeout", 86400000)
@shell.hbase.configuration.setInt("hbase.client.sync.wait.timeout.msec", 86400000)

7.5 导出快照时报错：IllegalStateException: Reached max limit of upload attempts for part 错误

本篇作者

• 耿立超（Laurence）