暂无图片
分享
Elan
2025-06-17
ogg 同步sqlserver 到kafka总是超时
源端sqlserver环境,OGG extract和pump正常,中间服务器replicat只要进程启动就异常退出,日志如下:replicat 参数:view params re_zyytj REPLICAT re_zyytj TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 5000 SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8") MAP dbo.*, TARGET dbo.*; view report 报错日志:2025-06-17 21:15:05 INFO OGG-06510 Using the following key columns for target table dbo.TS_IDX: IDX. 2025-06-17 21:15:05 INFO OGG-03010 Performing implicit conversion of column data from character set windows-936 to UTF-8. =ERROR 2025-06-17 21:17:05.000574 [kafka-producer-network-thread | producer-1] - A failure occurred sending a message to Kafka. org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ZJGZY-ZJGZYY_SYSTEM-2:120001 ms has passed since batch creation =ERROR 2025-06-17 21:17:05.000577 [main] - An exception occurred while trying to send a producer record to Kafka using the synchronous call. java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ZJGZY-ZJGZYY_SYSTEM-2:120001 ms has passed since batch creation at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:97) ~[kafka-clients-3.8.1.jar:?] at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:65) ~[kafka-clients-3.8.1.jar:?] at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:30) ~[kafka-clients-3.8.1.jar:?] at oracle.goldengate.handler.kafka.impl.BlockingKafkaProducer.send(BlockingKafkaProducer.java:47) [ggkafka-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.handler.kafka.KafkaHandler.processData(KafkaHandler.java:508) [ggkafka-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.handler.kafka.KafkaHandler.operationAdded(KafkaHandler.java:200) [ggkafka-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.datasource.DsEventManager$4.send(DsEventManager.java:367) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.datasource.DsEventManager.distributeEvent(DsEventManager.java:110) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.datasource.DsEventManager.fireOperationAdded(DsEventManager.java:376) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.datasource.AbstractDataSource.fireOperationAdded(AbstractDataSource.java:481) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.datasource.UserExitDataSource.addOperationToTransactionAndFireEvent(UserExitDataSource.java:2031) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.00 5] at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1534) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005] at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1318) [ggdbutil-12.3.2.1.1.005.jar:12.3.2.1.1.005] Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ZJGZY-ZJGZYY_SYSTEM-2:120001 ms has passed since batch creation Exception in thread "main" oracle.goldengate.util.GGException: Error detected handling operation added event. at oracle.goldengate.datasource.UserExitDataSource.addOperationToTransactionAndFireEvent(UserExitDataSource.java:2038) at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1534) at oracle.goldengate.datasource.UserExitDataSource.createOperation(UserExitDataSource.java:1318) Source Context : SourceModule : [gglib.ggdal.adapter.java] SourceID : [/scratch/aime/adestore/views/aime_adc4150560/oggcore/OpenSys/src/gglib/ggdal/Adapter/Java/JavaAdapter.cpp] SourceMethod : [HandleJavaException] SourceLine : [246] ThreadBacktrace : [21] elements : [/ogg/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f7838b4e0ae]] : [/ogg/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x6ac) [0x7f7838b3e9bc]] : [/ogg/libgglog.so(_MSG_String(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x39) [0x7f7838b2cd19]] : [/ogg/libggjava.so(+0x2e9e7) [0x7f78301969e7]] : [/ogg/libggjava.so(ggs::gglib::ggdal::CJavaRecordWriter::WriteInByteBuffer(ggs::gglib::ggdal::CDALRecord const&, int&, bool&, ggs::gglib::ggd al::CDALError&)+0x245) [0x7f783019c955]] : [/ogg/replicat(ggs::gglib::ggdal::CDALRecordWriter::Write(ggs::gglib::ggdal::CDALRecord const&, int&, bool&, ggs::gglib::ggdal::CDALError&)+0 x4a) [0x82edea]] : [/ogg/replicat(GenericImpl::Write(ObjectMetadata*, std_rec_hdr_def const*, ggs::gglib::ggdal::CDALRecord&)+0x52) [0x816832]] : [/ogg/replicat(execute_statement(ggs::gglib::gglcr::CommonLCR const*, char*, ObjectMetadata*, short, short*, short, extr_ptr_def*, int*, char *, unsigned long, void*, sql_stmt_params_t*)+0x245) [0x80e4f5]] : [/ogg/replicat(perform_sql_statements(ObjectMetadata*, ggs::gglib::gglcr::CommonLCR const*, short, char*, short*, short, extr_ptr_def*)+0xcfa ) [0x6eeeea]] : [/ogg/replicat(ggs::er::ReplicatContext::replicate_io(ggs::gglib::gglcr::CommonLCR const*, ggs::gglib::gglcr::CommonLCR*, extr_ptr_def*, bool , int&, RepCsn&, bool*)+0x10de) [0x6ea04e]] : [/ogg/replicat(ggs::er::ReplicatContext::processRecord(ggs::gglib::gglcr::CommonLCR const*, ggs::gglib::gglcr::CommonLCR*, extr_ptr_def*&, ex tr_ptr_def*&, bool&, int&, bool, RepCsn&)+0x318) [0x6f98d8]] : [/ogg/replicat(ggs::er::ReplicatContext::processReplicatLoop()+0x4614) [0x631494]] : [/ogg/replicat(ggs::er::ReplicatContext::run()+0x45) [0x610545]] : [/ogg/replicat() [0x6e506f]] : [/ogg/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x5e) [0x7e2d8e]] : [/ogg/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x173) [0x7e7153]] : [/ogg/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x140) [0x7e79c0]] : [/ogg/replicat(main+0x3b) [0x6e7e0b]] : [/usr/lib64/libc.so.6(+0x2d210) [0x7f78317d0210]] : [/usr/lib64/libc.so.6(__libc_start_main+0x7c) [0x7f78317d02bc]] : [/ogg/replicat() [0x550831]] 2025-06-17 21:17:05 ERROR OGG-15051 Java or JNI exception: oracle.goldengate.util.GGException: Error detected handling operation added event.投递表数据量约为300多张,kafka连接参数配置如下:cat kafka.props gg.handlerlist=kafkahandler gg.handler.kafkahandler.type=kafka gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties gg.handler.kafkahandler.topicMappingTemplate=ZJGZY-ZJGZYY_SYSTEM gg.handler.kafkahandler.format=json gg.handler.kafkahandler.mode=op gg.handler.kafkahandler.format.metaColumnsTemplate=${table},${optype},${timestamp},${currenttimestamp},${position},${alltokens} gg.handler.kafkahandler.BlockingSend=true gg.handler.kafkahandler.format.includePrimaryKeys=true gg.handler.kafkaconnect.keyMappingTemplate=${primaryKeys} goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec gg.classpath=/ogg/dirprm/:/usr/local/kafka/libs/*:/ogg/lib/* javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar cat custom_kafka_producer.propertiesbootstrap.servers=192.168.0.100:9092 acks=1 reconnect.backoff.ms=1000 value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer batch.size=16384 linger.ms=0 max.request.size = 5024000 send.buffer.bytes = 5024000 security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="password";
收藏
分享
1条回答
默认
最新
Elan
问题已关闭: 格式问题
暂无图片 评论
暂无图片 有用 0
回答交流
提交
问题信息
请登录之后查看
邀请回答
暂无人订阅该标签,敬请期待~~
暂无图片墨值悬赏