单机搭建Kafka集群

单机版kafka集群有什么作用

练习上手用。

搭建zookeeper集群

• 首先下载zookeeper

[apache zookeeper官网](https://zookeeper.apache.org/)

[apache zookeeper下载地址](http://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/)

[apache zookeeper 3.5.5.tar.gz](https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz)

• 解压 apache zookeeper
  

tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz

• 将zookeeper复制三份,分别命名为zookeeper-1,zookeeper-2,zookeeper-3

• 将zookeeper-1中的zoo.example.cfg文件复制一份改名为: zoo.cfg

• 修改config/zoo.cfg文件

  server.1=localhost.:2887:3887
  server.2=localhost.:2888:3888
  server.3=localhost.:2889:3889
  
  
  admin.serverPort=8000
  

  完成的配置文件如下:

• 修改端口: clientPort=2181
  

• 修改数据目录: dataDir=/ashura/zookeeper-1/datalog
  

• 增加以下配置:

 server.1=localhost.:2887:3887
 server.2=localhost.:2888:3888
 server.3=localhost.:2889:3889

• 完整的配置如下:

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use tmp for storage, tmp here is just
# example sakes.
dataDir=/ashura/zookeeper-1/datalog
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=localhost.:2887:3887
server.2=localhost.:2888:3888
server.3=localhost.:2889:3889

• 将这份zoo.cfg
  分别复制到zookeeper-2
  ,zookeeper-3
  的config
  目录下.

• 修改zookeeper2
  的zoo.cfg
  中clientPort=2183
  ,dataDir=/ashura/zookeeper-2/datalog
  

• 修改zookeeper3
  的zoo.cfg
  中clientPort=2184
  ,dataDir=/ashura/zookeeper-3/datalog
  

• 创建刚才在配置文件中写的目录

mkdir ashura/zookeeper-1/datalog
mkdir ashura/zookeeper-2/datalog
mkdir ashura/zookeeper-3/datalog

• 分别 在datalog目录下 执行以下命令,写入myid。

echo "1" > /ashura/zookeeper-1/datalog/myid
echo "2" > /ashura/zookeeper-2/datalog/myid
echo "3" > /ashura/zookeeper-3/datalog/myid

• 最后分别启动zookeeper集群

/ashura/zookeeper-1/bin/zkServer.sh start
/ashura/zookeeper-2/bin/zkServer.sh start
/ashura/zookeeper-3/bin/zkServer.sh start

使用如下命令判断是否启动成功

/ashura/zookeeper-1/bin/zkServer.sh status
/ashura/zookeeper-2/bin/zkServer.sh status
/ashura/zookeeper-3/bin/zkServer.sh status

搭建Kafka集群

下载

[kafka官网](http://kafka.apache.org/)

[kafka_2.11-2.2.1.tgz](http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.1/kafka_2.11-2.2.1.tgz)

开始安装

• 解压

tar -zxvf kafka_2.11-2.2.1.tgz

• 将config/server.properties
  复制三份，分别命名为server1.properties
  ,server2.properties
  ,server3.properties
  。

• 修改server1.properties
  

• 
  

  broker.id=1
  listeners=PLAINTEXT://:9092
  advertised.listeners=PLAINTEXT://10.1.14.159:9092(其中10.1.14.159是我本机的ip)
  log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka1-logs
  zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
  

• 同理,修改server2.properties
  

broker.id=2
listeners=PLAINTEXT://:9093
advertised.listeners=PLAINTEXT://10.1.14.159:9093(其中10.1.14.159是我本机的ip)
log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka2-logs
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

• 同理,修改server3.properties
  

broker.id=3
listeners=PLAINTEXT://:9094
advertised.listeners=PLAINTEXT://10.1.14.159:9094(其中10.1.14.159是我本机的ip)
log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka3-logs
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

• 然后执行以下命令

nohup ashura/kafka_2.11-2.2.1/bin/kafka-server-start.sh ashura/kafka_2.11-2.2.1/config/server3.properties > ashura/kafka_2.11-2.2.1/logs/kafka3-logs/startup.log 2>&1 &
nohup ashura/kafka_2.11-2.2.1/bin/kafka-server-start.sh ashura/kafka_2.11-2.2.1/config/server2.properties > ashura/kafka_2.11-2.2.1/logs/kafka2-logs/startup.log 2>&1 &
nohup ashura/kafka_2.11-2.2.1/bin/kafka-server-start.sh ashura/kafka_2.11-2.2.1/config/server1.properties > ashura/kafka_2.11-2.2.1/logs/kafka1-logs/startup.log 2>&1 &

• 通过startup.log,或者同级目录下的server.log查看是否有报错即可。

• 检测

• 创建主题:./kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic fxb_test1 --replication-factor 3 --partitions 3
  

• 启动消费者: ./kafka-console-producer.sh --broker-list 10.1.14.159:9092 --topic fxb_test1
  

• 新开窗口个,启动生产者: kafka-console-producer.sh --bootstrap-server 127.0.0.1 --create --topic fxb_test1
  
  在生产者窗口中输入消息，查看消费者的窗口，是否有消息产生。

使用java client进行测试

• 引入依赖

<dependencies>
    <dependency>
        <groupId>org.apache.kafkagroupId>
        <artifactId>kafka-clientsartifactId>
        <version>2.3.0version>
    dependency>
dependencies>

• 创建生产者

package com.fxb.learn.kafka.producer;


import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;


import java.util.Properties;


/***
 * 生产者
 */
public class ProducerDemo {


    public static void main(String[] args) {


        Properties props = new Properties();
        props.put("bootstrap.servers", "10.127.138.75:9092");
        props.put("acks", "all");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//        props.put(ProducerConfig.Re)


        Producer<String, String> producer = new KafkaProducer<>(props);
        for (int i = 0; i < 10; i++) {
            producer.send(new ProducerRecord<String, String>("fxb_test1", Integer.toString(i), Integer.toString(i)));
            System.out.println("has sent msg [" + i + "]");
        }
        producer.close();
    }
}

• 创建消费者

package com.fxb.learn.kafka.consumer;


import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;


import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;


/***
 * 消费者
 */
public class ConsumerDemo {


    public static void main(String[] args) {
        Properties props = new Properties();
        props.setProperty("bootstrap.servers", "10.127.138.75:9092");
        props.setProperty("group.id", "test");
        props.setProperty("enable.auto.commit", "true");
        props.setProperty("auto.commit.interval.ms", "1000");
        props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("fxb_test1", "bar"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records)
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
        }
    }
}

参考文档

[Java DOC](https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html)