2025-06-17
ogg 同步sqlserver 到kafka总是超时
源端sqlserver环境,OGG extract和pump正常,中间服务器replicat只要进程启动就异常退出,日志如下:replicat 参数:view params re_zyytj
REPLICAT re_zyytj
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 5000
SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
MAP dbo.*, TARGET dbo.*;
view report 报错日志:2025-06-17 21:15:05 INFO OGG-06510 Using the following key columns for target table dbo.TS_IDX: IDX.
2025-06-17 21:15:05 INFO OGG-03010 Performing implicit conversion of column data from character set windows-936 to UTF-8.
=ERROR 2025-06-17 21:17:05.000574 [kafka-producer-network-thread | producer-1] - A failure occurred sending a message to Kafka.
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ZJGZY-ZJGZYY_SYSTEM-2:120001 ms has passed since batch creation
=ERROR 2025-06-17 21:17:05.000577 [main] - An exception occurred while trying to send a producer record to Kafka using the synchronous call.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ZJGZY-ZJGZYY_SYSTEM-2:120001 ms has passed since batch
creation
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:97) ~[kafka-clients-3.8.1.jar:?]
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:65) ~[kafka-clients-3.8.1.jar:?]
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:30) ~[kafka-clients-3.8.1.jar:?]
at oracle.goldengate.handler.kafka.impl.BlockingKafkaProducer.send(BlockingKafkaProducer.java:47) [ggkafka-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.handler.kafka.KafkaHandler.processData(KafkaHandler.java:508) [ggkafka-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.handler.kafka.KafkaHandler.operationAdded(KafkaHandler.java:200) [ggkafka-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.datasource.DsEventManager$4.send(DsEventManager.java:367) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.datasource.DsEventManager.distributeEvent(DsEventManager.java:110) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.datasource.DsEventManager.fireOperationAdded(DsEventManager.java:376) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.datasource.AbstractDataSource.fireOperationAdded(AbstractDataSource.java:481) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.datasource.UserExitDataSource.addOperationToTransactionAndFireEvent(UserExitDataSource.java:2031) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.00
5]
at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1534) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005]
at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1318) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ZJGZY-ZJGZYY_SYSTEM-2:120001 ms has passed since batch creation
Exception in thread "main" oracle.goldengate.util.GGException: Error detected handling operation added event.
at oracle.goldengate.datasource.UserExitDataSource.addOperationToTransactionAndFireEvent(UserExitDataSource.java:2038)
at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1534)
at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1318)
Source Context :
SourceModule : [gglib.ggdal.adapter.java]
SourceID : [/scratch/aime/adestore/views/aime_adc4150560/oggcore/OpenSys/src/gglib/ggdal/Adapter/Java/JavaAdapter.cpp]
SourceMethod : [HandleJavaException]
SourceLine : [246]
ThreadBacktrace : [21] elements
: [/ogg/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f7838b4e0ae]]
: [/ogg/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x6ac) [0x7f7838b3e9bc]]
: [/ogg/libgglog.so(_MSG_String(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x39) [0x7f7838b2cd19]]
: [/ogg/libggjava.so(+0x2e9e7) [0x7f78301969e7]]
: [/ogg/libggjava.so(ggs::gglib::ggdal::CJavaRecordWriter::WriteInByteBuffer(ggs::gglib::ggdal::CDALRecord const&, int&, bool&, ggs::gglib::ggd
al::CDALError&)+0x245) [0x7f783019c955]]
: [/ogg/replicat(ggs::gglib::ggdal::CDALRecordWriter::Write(ggs::gglib::ggdal::CDALRecord const&, int&, bool&, ggs::gglib::ggdal::CDALError&)+0
x4a) [0x82edea]]
: [/ogg/replicat(GenericImpl::Write(ObjectMetadata*, std_rec_hdr_def const*, ggs::gglib::ggdal::CDALRecord&)+0x52) [0x816832]]
: [/ogg/replicat(execute_statement(ggs::gglib::gglcr::CommonLCR const*, char*, ObjectMetadata*, short, short*, short, extr_ptr_def*, int*, char
*, unsigned long, void*, sql_stmt_params_t*)+0x245) [0x80e4f5]]
: [/ogg/replicat(perform_sql_statements(ObjectMetadata*, ggs::gglib::gglcr::CommonLCR const*, short, char*, short*, short, extr_ptr_def*)+0xcfa
) [0x6eeeea]]
: [/ogg/replicat(ggs::er::ReplicatContext::replicate_io(ggs::gglib::gglcr::CommonLCR const*, ggs::gglib::gglcr::CommonLCR*, extr_ptr_def*, bool
, int&, RepCsn&, bool*)+0x10de) [0x6ea04e]]
: [/ogg/replicat(ggs::er::ReplicatContext::processRecord(ggs::gglib::gglcr::CommonLCR const*, ggs::gglib::gglcr::CommonLCR*, extr_ptr_def*&, ex
tr_ptr_def*&, bool&, int&, bool, RepCsn&)+0x318) [0x6f98d8]]
: [/ogg/replicat(ggs::er::ReplicatContext::processReplicatLoop()+0x4614) [0x631494]]
: [/ogg/replicat(ggs::er::ReplicatContext::run()+0x45) [0x610545]]
: [/ogg/replicat() [0x6e506f]]
: [/ogg/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x5e) [0x7e2d8e]]
: [/ogg/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x173) [0x7e7153]]
: [/ogg/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x140) [0x7e79c0]]
: [/ogg/replicat(main+0x3b) [0x6e7e0b]]
: [/usr/lib64/libc.so.6(+0x2d210) [0x7f78317d0210]]
: [/usr/lib64/libc.so.6(__libc_start_main+0x7c) [0x7f78317d02bc]]
: [/ogg/replicat() [0x550831]]
2025-06-17 21:17:05 ERROR OGG-15051 Java or JNI exception:
oracle.goldengate.util.GGException: Error detected handling operation added event.投递表数据量约为300多张,kafka连接参数配置如下:cat kafka.props
gg.handlerlist=kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
gg.handler.kafkahandler.topicMappingTemplate=ZJGZY-ZJGZYY_SYSTEM
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format.metaColumnsTemplate=${table},${optype},${timestamp},${currenttimestamp},${position},${alltokens}
gg.handler.kafkahandler.BlockingSend=true
gg.handler.kafkahandler.format.includePrimaryKeys=true
gg.handler.kafkaconnect.keyMappingTemplate=${primaryKeys}
goldengate.userexit.timestamp=utc
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
gg.classpath=/ogg/dirprm/:/usr/local/kafka/libs/*:/ogg/lib/*
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar
cat custom_kafka_producer.propertiesbootstrap.servers=192.168.0.100:9092
acks=1
reconnect.backoff.ms=1000
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
batch.size=16384
linger.ms=0
max.request.size = 5024000
send.buffer.bytes = 5024000
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="password";
收藏
分享
1条回答
默认
最新
回答交流
提交
问题信息
请登录之后查看
邀请回答
暂无人订阅该标签,敬请期待~~
墨值悬赏

评论
