暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

OGG海量数据分片同步到tidb

朱泰伯 2025-02-13
148

一、抽取进程

1.1 创建抽取进程

EXTRACT E_OR


setenv (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
USERID ogg@scott, PASSWORD "112233"
EXTTRAIL /scott/ogg19c/dirdat/or
IGNOREREPLICATES
GETAPPLOPS
TRANLOGOPTIONS EXCLUDEUSER OGG
TABLE  customer.ORDER, COLSEXCEPT(ID);

1.2、增加抽取进程

add extract E_OR tranlog, INTEGRATED tranlog, begin now
add exttrail /scott/ogg19c/dirdat/or extract E_CAPOR, MEGABYTES 500
register EXTRACT E_OR, DATABASE
start E_OR


二、投递进程

2.1 创建投递进程

EXTRACT P_OR

setenv (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
RMTHOST 192.167.253.137, MGRPORT 7808
RMTTRAIL /scott/ogg21cl/dirdat/or

--tables
table Customer.TORDER;

2.2 增加进程

add extract P_OR, exttrailsource /scott/ogg19c/dirdat/or
add rmttrail /scott/ogg21c/dirdat/or, extract P_OR, MEGABYTES 500


三,复制进程

3.1分片1

replicat REP_01
sourcecharset al32utf8
container mem_limit 8G
cachemgr cachesize 1G
discardfile /scott/ogg21c/dirrpt/REP_01.dsc, purge
targetdb tidb@192.167.1.138:4000
userid ogg password "112233"
SQLEXEC "set @@allow_auto_random_explicit_insert = true"
ASSUMETARGETDEFS
--HANDLECOLLISIONS
MAP Customer.ORDER, TARGET Customer.ORDER, KEYCOLS(NO),
FILTER(@RANGE(1,9,NO));
3.1 分片2
replicat REP_02
sourcecharset al32utf8
container mem_limit 8G
cachemgr cachesize 1G
discardfile /data/ogg21c_mysql/dirrpt/REP_C02.dsc, purge
targetdb tidb@192.167.1.138:4000
userid ogg password "112233"
SQLEXEC "set @@allow_auto_random_explicit_insert = true"
ASSUMETARGETDEFS
--HANDLECOLLISIONS
MAP Customer.ORDER, TARGET Customer.ORDER, KEYCOLS(NO),
FILTER(@RANGE(2,9,NO));


3.3 增加负责进程

add replicat REP_C01, exttrail /scott/ogg21c/dirdat/or, checkpointtable Customer.oggchkpt
add replicat REP_C02, exttrail /scott/ogg21c/dirdat/or, checkpointtable Customer.oggchkpt


四、初始化

4.1 分片1

extract ini_01

userid ogg@scott password "112233"
rmthost 192.168.167.137, mgrport 7808
rmttask replicat, group sr_01
reportcount every 60 seconds, rate

--tables
table Customer.ORDER, COLSEXCEPT(ID), FILTER (@RANGE(1, 100, NO));


4.1 初始化复制进程分片1 (注意负责进程的名字在抽取初始化进程里已指定)

replicat sr_01

setenv (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
--SOURCEDEFS ./dirdef/def_ucap.def

ASSUMETARGETDEFS

targetdb tidb@192.168.1.138:4000
userid ogg password "112233"

--CHECKPOINTTABLE ogg.ggschkpt

SQLEXEC "set tidb_constraint_check_in_place=1"
SQLEXEC "set @@allow_auto_random_explicit_insert = true"
DISCARDFILE ./dirrpt/sr_mtr1.dsc, purge

cachemgr cachesize 1G
container mem_limit 2G

--HANDLECOLLISIONS

--tables
map Customer.ORDER, target Customer.ORDER, KEYCOLS(ID);


4.3 增加初始化进程(100个分片都要加)

ADD REPLICAT sr_01, SPECIALRUN

dblogin userid ogg@scott, password "112233"
ADD EXTRACT ini_01, SOURCEISTABLE

五、启动
先启动抽取进程

E_OR

然后启动初始化抽取进程ini_01 (100个都要启动)

启动之后初始化复制进程自动启动了。


「喜欢这篇文章,您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容,转载时必须标注文章的来源(墨天轮),文章链接,文章作者等基本信息,否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容,欢迎发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

评论