TiDB Lightning

TiDB Lightning

TiDB Lightning 是一个将全量数据高速导入到 TiDB 集群的工具，目前支持 SQL 或 CSV 输出格式的数据源。

1. 下载 TiDB Lightning 安装包

wget https://download.pingcap.org/tidb-toolkit-V5.0.0-rc-linux-amd64.tar.gz

tar zxvf tidb-toolkit-v5.0.0-rc-linux-amd64.tar.gz

cd opt/tidb-toolkit-v5.0.0-rc-linux-amd64

2. 源库导出（MySQL）

./dumpling -h 172.26.55.21 -u root -P 3306 -p mysql  -t 16 -F 256MB  -o data/mysql

• -B test：从 test 数据库导出

• -f test.t[12]：只导出 test.t1 和 test.t2 这两个表

• -t 16：使用 16 个线程导出数据

• -F 256MB：将每张表切分成多个文件，每个文件大小约为 256 MB

• -o data/mysql：导出目录

3. 配置 toml 文件

vim opt/tidb-toolkit-v5.0.0-rc-linux-amd64/bin/tidb-lightning.toml

[lightning]
# 日志
level = "info"
file = "tidb-lightning.log"

[tikv-importer]
# 选择使用的 local 后端
backend = "local"
# 设置排序的键值对的临时存放地址，目标路径需要是一个空目录
"sorted-kv-dir" = "/mnt/ssd/sorted-kv-dir"

[mydumper]
# 源数据目录。
data-source-dir = "/data/mysql/"

# 配置通配符规则，默认规则会过滤 mysql、sys、INFORMATION_SCHEMA、PERFORMANCE_SCHEMA、METRICS_SCHEMA、INSPECTION_SCHEMA 系统数据库下的所有表
# 若不配置该项，导入系统表时会出现“找不到 schema”的异常
filter = ['*.*', '!mysql.*', '!sys.*', '!INFORMATION_SCHEMA.*', '!PERFORMANCE_SCHEMA.*', '!METRICS_SCHEMA.*', '!INSPECTION_SCHEMA.*']
[tidb]
# 目标集群的信息
host = "172.26.55.91"
port = 4000
user = "root"
password = "123456"
# 表架构信息在从 TiDB 的“状态端口”获取。
status-port = 10080
# 集群 pd 的地址
pd-addr = "172.26.55.91:2379"

4. 导入数据

如果直接在命令行中用 nohup 启动程序，可能会因为 SIGHUP 信号而退出，建议把 nohup 放到脚本里面：

vim mysql2tidb.sh

#!/bin/bash
nohup ./tidb-lightning -config tidb-lightning.toml > nohup.out &

执行脚本：

sh mysql2tidb.sh

查看日志：

tailf -n 100 tidb-lightning.log

后记

前几次测试 Lightning 组件都没有报错，但在验证业务 sql 时发现部分表没有数据。之后重新通过 dumpling 导出加 tidb-lightning 导入的方式补齐数据。

但出现了一下报错：

[2021/03/29 15:36:39.386 +08:00] [ERROR] [restore.go:761] ["TiDB Lightning has failed last time. To prevent data loss, this run will stop now. Please resolve errors first"] [count=12]

[2021/03/29 15:36:39.391 +08:00] [ERROR] [restore.go:290] ["run failed"] [step=3] [error="TiDB Lightning has failed last time; please resolve these errors first"]
[2021/03/29 15:36:39.391 +08:00] [ERROR] [restore.go:301] ["the whole procedure failed"] [takeTime=832.220511ms] [error="TiDB Lightning has failed last time; please resolve these errors fir
st"]
[2021/03/29 15:36:39.391 +08:00] [ERROR] [main.go:91] ["tidb lightning encountered error stack info"] [error="TiDB Lightning has failed last time; please resolve these errors first"] [error
Verbose="TiDB Lightning has failed last time; please resolve these errors first\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).restoreTables\n\t/home/jenkins/agen
t/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:791\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*RestoreControl
ler).Run\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:277\ngithub.com/pingcap/tidb-lightning/ligh
tning.(*Lightning).run\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:310\ngithub.com/pingcap/tidb-lightn
ing/lightning.(*Lightning).RunOnce\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:191\nmain.main.func2\n\
t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning/main.go:86\nmain.main\n\t/home/jenkins/agent/workspace/optimizat
ion-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning/main.go:88\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/r
untime/asm_amd64.s:1357"]
[2021/03/29 15:36:39.391 +08:00] [ERROR] [main.go:92] ["tidb lightning encountered error"] [error="TiDB Lightning has failed last time; please resolve these errors first"]

就这些输出来说，并没有输出具体的错误。后续将研究并测试 Data Migration 组件，官方的意思是 DM 更稳定而且功能更强。