Zookeeper+Kafka叢集部署
阿新 • • 發佈：2018-12-21
主機規劃：
10.200.3.85 Kafka+ZooKeeper
10.200.3.86 Kafka+ZooKeeper
10.200.3.87 Kafka+ZooKeeper
軟體下載地址：
#wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz #wget http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz
三臺主機hosts檔案一致：
# cat /etc/hosts 10.200.3.85 ZooKeeper-Kafka-01 10.200.3.86 ZooKeeper-Kafka-02 10.200.3.87 ZooKeeper-Kafka-03
一、安裝zookeeper
1.在master節點上操作：
[[email protected] src]# tar -zxvf zookeeper-3.4.10.tar.gz -C /usr/local/ [[email protected] src]# cd .. [[email protected] local]# ln -s zookeeper-3.4.10 zookeeper [[email protected] local]# cd zookeeper/conf/ [[email protected] conf]# cp zoo_sample.cfg zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/tmp/zookeeper clientPort=2181 server.1=ZooKeeper-Kafka-01:2888:3888 server.2=ZooKeeper-Kafka-02:2888:3888 server.3=ZooKeeper-Kafka-03:2888:3888
2.建立dataDir目錄建立/tmp/zookeeper
# 在master節點上
[[email protected] conf]# mkdir /tmp/zookeeper [[email protected] conf]# touch /tmp/zookeeper/myid [[email protected] conf]# echo 1 > /tmp/zookeeper/myid
3.將zookeeper檔案複製到另外兩個節點：
[[email protected] local]# scp -r zookeeper-3.4.10/ 10.200.3.86:/usr/local/ [[email protected]per-Kafka-01 local]# scp -r zookeeper-3.4.10/ 10.200.3.87:/usr/local/
4.在兩個slave節點建立目錄和檔案
#ZooKeeper-Kafka-02節點:
[[email protected] local]# ln -s zookeeper-3.4.10 zookeeper [[email protected] local]# mkdir /tmp/zookeeper [[email protected] local]# touch /tmp/zookeeper/myid [[email protected] local]# echo 2 > /tmp/zookeeper/myid
#ZooKeeper-Kafka-03節點
[[email protected] local]# ln -s zookeeper-3.4.10 zookeeper [[email protected] local]# mkdir /tmp/zookeeper [[email protected] local]# touch /tmp/zookeeper/myid [[email protected] local]# echo 3 > /tmp/zookeeper/myid
5.分別在每個節點上啟動 zookeeper測試：
[[email protected] zookeeper]# ./bin/zkServer.sh start [[email protected] zookeeper]# ./bin/zkServer.sh start [[email protected] zookeeper]# ./bin/zkServer.sh start
6.檢視狀態：
[[email protected] zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: follower
[[email protected] zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: leader
[[email protected] zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: follower
至此zookeeper叢集安裝成功！！！
Kafka叢集安裝配置
1.安裝：
[[email protected] src]# tar -zxvf kafka_2.12-1.1.0.tgz -C /usr/local/ [[email protected] src]# cd .. [[email protected] local]# ln -s kafka_2.12-1.1.0 kafka
2.修改server.properties檔案
# master為0 broker.id=0 listeners=PLAINTEXT://ZooKeeper-Kafka-01:9092 advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-01:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=5 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=24 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 # 連線 zookeeper.connect=ZooKeeper-Kafka-01:2181,ZooKeeper-Kafka-02:2181,ZooKeeper-Kafka-03:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 # 可刪除 topic delete.topic.enable=true
3.將 kafka_2.12-1.1.0 資料夾複製到另外兩個節點下
[[email protected] local]# scp -r kafka_2.12-1.1.0/ 10.200.3.86:/usr/local/ [[email protected] local]# scp -r kafka_2.12-1.1.0/ 10.200.3.87:/usr/local/
並修改每個節點對應的 server.properties 檔案的 broker.id和listenrs：
[[email protected] config]# cat server.properties broker.id=1 listeners=PLAINTEXT://ZooKeeper-Kafka-02:9092 advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-02:9092 [[email protected] config]# cat server.properties broker.id=2 listeners=PLAINTEXT://ZooKeeper-Kafka-03:9092 advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-03:9092
4.啟動服務
bin/kafka-server-start.sh config/server.properties &
Zookeeper+Kafka叢集測試
建立topic：
[[email protected] kafka]# bin/kafka-topics.sh --create --zookeeper ZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 --replication-factor 3 --partitions 3 --topic test
顯示topic：
[[email protected] kafka]# bin/kafka-topics.sh --describe --zookeeper ZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 --topic test
列出topic：
[[email protected] kafka]# bin/kafka-topics.sh --list --zookeeper ZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 test
建立 producer(生產者);
# 在master節點上 測試生產訊息
[[email protected] kafka]# bin/kafka-console-producer.sh --broker-list ZooKeeper-Kafka-01:9092 -topic test >hello world >[2018-04-03 12:18:25,545] INFO Updated PartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} for Partition: test-0. Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache) this is example ... >[2018-04-03 12:19:16,342] INFO Updated PartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} for Partition: test-2. Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache) welcome to china >[2018-04-03 12:20:53,141] INFO Updated PartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} for Partition: test-1. Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache)
建立 consumer(消費者):
# 在ZooKeeper-Kafka-02節點上 測試消費
[[email protected] kafka]# bin/kafka-console-consumer.sh --zookeeper ZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 -topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
this is example ...
hello world
[2018-04-03 12:20:53,145] INFO Updated PartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset:-1} for Partition: test-1. Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache)
welcome to china
#在ZooKeeper-Kafka-03節點上 測試消費
[[email protected] kafka]# bin/kafka-console-consumer.sh --zookeeper ZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 -topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. welcome to china hello world this is example ...
然後在 producer 裡輸入訊息，consumer 中就會顯示出同樣的內容，表示消費成功
刪除 topic 和關閉服務
[[email protected] kafka]# bin/kafka-topics.sh --delete --zookeeper ZooKeeper-Kafka-01:2181, ZooKeeper-Kafka-02:2181, ZooKeeper-Kafka-03:2181 --topic test
啟動服務:
bin/kafka-server-start.sh config/server.properties &
停止服務：
bin/kafka-server-stop.sh
至此Zookeeper+Kafka叢集配置成功.