暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

kafka connect应用之:文本数据的接入接出

数据的砖格外烫手 2018-06-05
455


source数据源配置:

vagrant@vagrant-ubuntu-trusty-64:/mnt/etc$ cat connect-file-source.properties 

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.


name=local-file-source

connector.class=FileStreamSource

tasks.max=1

file=test.txt

topic=file_connect_test


sink到文件的配置:

vagrant@vagrant-ubuntu-trusty-64:/mnt/etc$ cat connect-file-sink.properties 

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.


name=local-file-sink

connector.class=FileStreamSink

tasks.max=1

file=test.sink.txt

topics=file_connect_test


connect-standalone :加载配置文件

vagrant@vagrant-ubuntu-trusty-64:~$ cat conn_file.sh 

nohup connect-standalone /mnt/etc/connect-avro-standalone.properties /mnt/etc/connect-file-source.properties /mnt/etc/connect-file-sink.properties  > conn_file.log 2>&1 & 


tail -f conn_file.log 


验证:kafka消费:

vagrant@vagrant-ubuntu-trusty-64:~$ kafka-console-consumer  --zookeeper localhost:2181 --topic file_connect_test  --from-beginning


dadada

log line 1

log line 2

log line 3

log line 1

log line 2

log line 3

0log line 1 for nico demo

0log line 2 for nico demo



验证:sink到文件:


vagrant@vagrant-ubuntu-trusty-64:~$ tail test.sink.txt 

log line 91 for nico demo

log line 92 for nico demo

log line 93 for nico demo

log line 94 for nico demo

log line 95 for nico demo

log line 96 for nico demo

log line 97 for nico demo

log line 98 for nico demo

log line 99 for nico demo


注意事项:

  1. properties文件中的 topic 要一致:source 文件和 sink文件要对应。

  2. properties文件中,如果不指定绝对路径,默认的根目录是 ~/ , 本文中是

vagrant@vagrant-ubuntu-trusty-64:~$ cd

vagrant@vagrant-ubuntu-trusty-64:~$ pwd

/home/vagrant

vagrant@vagrant-ubuntu-trusty-64:~$ 



参考资料:

[1] https://gerardnico.com/dit/kafka/connector/file-source


[2] https://docs.confluent.io/current/connect/connect-filestream/filestream_connector.html




文章转载自数据的砖格外烫手,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

评论