Zookeeper+Kafka集群部署

IT那活儿 2025-12-01

点击上方“IT那活儿”公众号--专注于企业全栈运维技术分享，不管IT什么活儿，干就完了！！！

一

部署背景

1.1 部署原因

因业务扩展需部署Zookeeper+Kafka集群。

1.2 主机规划

192.195.30.193 　　Kafka+ZooKeeper
192.195.30.194 　　Kafka+ZooKeeper
192.195.30.195 　　 Kafka+ZooKeeper

1.3 软件下载地址

#wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

#wget http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz

1.4 三台主机hosts文件一致

# cat etc/hosts
192.195.30.193 ZooKeeper-Kafka-01 
192.195.30.194 ZooKeeper-Kafka-02
192.195.30.195 ZooKeeper-Kafka-03

二

安装zookeeper

2.1 在master节点上操作

[root@ZooKeeper-Kafka-01 src]# tar -zxvf zookeeper-3.4.10.tar.gz -C usr/local/
[root@ZooKeeper-Kafka-01 src]# cd ..
[root@ZooKeeper-Kafka-01 local]# ln -s zookeeper-3.4.10 zookeeper
[root@ZooKeeper-Kafka-01 local]# cd zookeeper/conf/
[root@ZooKeeper-Kafka-01 conf]# cp zoo_sample.cfg zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
server.1=ZooKeeper-Kafka-01:2888:3888
server.2=ZooKeeper-Kafka-02:2888:3888
server.3=ZooKeeper-Kafka-03:2888:3888

2.2 创建dataDir目录创建/tmp/zookeeper

# 在master节点上：

[root@ZooKeeper-Kafka-01 conf]# mkdir tmp/zookeeper
[root@ZooKeeper-Kafka-01 conf]# touch tmp/zookeeper/myid
[root@ZooKeeper-Kafka-01 conf]# echo 1 > tmp/zookeeper/myid

2.3 将zookeeper文件复制到另外两个节点

[root@ZooKeeper-Kafka-01 local]# scp -r zookeeper-3.4.10/ 10.200.3.86:/usr/local/
[root@ZooKeeper-Kafka-01 local]# scp -r zookeeper-3.4.10/ 10.200.3.87:/usr/local/

2.4 在两个slave节点创建目录和文件

#ZooKeeper-Kafka-02节点：

[root@ZooKeeper-Kafka-02 local]# ln -s zookeeper-3.4.10 zookeeper
[root@ZooKeeper-Kafka-02 local]# mkdir tmp/zookeeper
[root@ZooKeeper-Kafka-02 local]# touch tmp/zookeeper/myid
[root@ZooKeeper-Kafka-02 local]# echo 2 > tmp/zookeeper/myid

#ZooKeeper-Kafka-03节点：

[root@ZooKeeper-Kafka-03 local]# ln -s zookeeper-3.4.10 zookeeper
[root@ZooKeeper-Kafka-03 local]# mkdir tmp/zookeeper
[root@ZooKeeper-Kafka-03 local]# touch tmp/zookeeper/myid
[root@ZooKeeper-Kafka-03 local]# echo 3 > tmp/zookeeper/myid

2.5 分别在每个节点上启动 zookeeper测试

[root@ZooKeeper-Kafka-01 zookeeper]# ./bin/zkServer.sh start
[root@ZooKeeper-Kafka-02 zookeeper]# ./bin/zkServer.sh start
[root@ZooKeeper-Kafka-03 zookeeper]# ./bin/zkServer.sh start

2.6 查看状态

[root@ZooKeeper-Kafka-01 zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled bydefault
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@ZooKeeper-Kafka-02 zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled bydefault
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[root@ZooKeeper-Kafka-03 zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled bydefault
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower

至此zookeeper集群安装成功！！！

三

Kafka集群安装配置

3.1 安装

[root@ZooKeeper-Kafka-01 src] # tar -zxvf kafka_2.12-1.1.0.tgz -C /usr/local/
[root@ZooKeeper-Kafka-01 src]# cd ..
[root@ZooKeeper-Kafka-01 local]# ln -s kafka_2.12-1.1.0 kafka

3.2 修改server.properties文件

# master为0
broker.id=0
listeners=PLAINTEXT://ZooKeeper-Kafka-01:9092
advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-01:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=5
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
# 连接
zookeeper.connect=ZooKeeper-Kafka-01:2181,ZooKeeper-Kafka-02:2181,ZooKeeper-Kafka-03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 可删除topic
delete.topic.enable=true

3.3 将 kafka_2.12-1.1.0 文件夹复制到另外两个节点下

[root@ZooKeeper-Kafka-01 local]# scp -r kafka_2.12-1.1.0/ 10.200.3.86:/usr/local/
[root@ZooKeeper-Kafka-01 local]# scp -r kafka_2.12-1.1.0/ 10.200.3.87:/usr/local/

并修改每个节点对应的 server.properties 文件的 broker.id和listenrs：

[root@ZooKeeper-Kafka-02 config]# cat server.properties 
broker.id=1
listeners=PLAINTEXT://ZooKeeper-Kafka-02:9092
advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-02:9092

[root@ZooKeeper-Kafka-03 config]# cat server.properties
broker.id=2
listeners=PLAINTEXT://ZooKeeper-Kafka-03:9092
advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-03:9092

3.4 启动服务

bin/kafka-server-start.sh config/server.properties &

四

Zookeeper+Kafka集群测试

4.1 创建topic

[root@ZooKeeper-Kafka-01 kafka]# bin/kafka-topics.sh--create--zookeeperZooKeeper-Kafka-01:2181, 
ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181--replication-factor3--partitions3--topictest

显示topic：

[root@ZooKeeper-Kafka-01 kafka]# bin/kafka-topics.sh--describe--zookeeperZooKeeper-Kafka-01:2181, 
ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181--topictest

列出topic：

[root@ZooKeeper-Kafka-01 kafka]# bin/kafka-topics.sh--list--zookeeperZooKeeper-Kafka-01:2181, 
ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181
test

4.2 创建 producer(生产者)

# 在master节点上测试生产消息：

[root@ZooKeeper-Kafka-01 kafka]# bin/kafka-console-producer.sh--broker-listZooKeeper-Kafka-01:9092-topictest
>helloworld
>[2018-04-03 12:18:25,545]INFOUpdatedPartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} forPartition: test-0. Cachenowcontains0entries. (kafka.server.epoch.LeaderEpochFileCache)
thisisexample ...
>[2018-04-03 12:19:16,342]INFOUpdatedPartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} forPartition: test-2. Cachenowcontains0entries. (kafka.server.epoch.LeaderEpochFileCache)
welcometochina
>[2018-04-03 12:20:53,141]INFOUpdatedPartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} forPartition: test-1. Cachenowcontains0entries. (kafka.server.epoch.LeaderEpochFileCache)

4.3 创建 consumer(消费者)

# 在ZooKeeper-Kafka-02节点上测试消费：

[root@ZooKeeper-Kafka-02 kafka]# bin/kafka-console-consumer.sh--zookeeperZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181-topictest--from-beginning
UsingtheConsoleConsumerwitholdconsumerisdeprecatedandwillberemovedinafuturemajorrelease. Considerusingthenewconsumerbypassing[bootstrap-server]insteadof[zookeeper].
thisisexample ...
hello world
[2018-04-03 12:20:53,145]INFOUpdatedPartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} forPartition: test-1. Cachenowcontains0entries. (kafka.server.epoch.LeaderEpochFileCache)
welcome to chins

#在ZooKeeper-Kafka-03节点上测试消费：

[root@ZooKeeper-Kafka-03 kafka]#  bin/kafka-console-consumer.sh --zookeeper ZooKeeper-Kafka-01:2181, 
ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 -topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. 
Considerusing the new consumer bypassing [bootstrap-server] instead of [zookeeper].
welcome to china
hello world
this is example ...

然后在 producer 里输入消息，consumer 中就会显示出同样的内容，表示消费成功。

4.4 删除 topic 和关闭服务

[root@ZooKeeper-Kafka-01 kafka]# bin/kafka-topics.sh--delete--zookeeperZooKeeper-Kafka-01:2181, 
ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181--topictest

启动服务:

bin/kafka-server-start.sh config/server.properties &

停止服务：

bin/kafka-server-stop.sh

至此Zookeeper+Kafka集群配置成功！

END

本文作者：陈竹彬(上海新炬中北团队）

本文来源：“IT那活儿”公众号

文章转载自IT那活儿，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

Zookeeper+Kafka集群部署

本文作者：陈竹彬(上海新炬中北团队）

本文来源：“IT那活儿”公众号

评论