Hadoop生態圈-Kafka的完全分布式部署
阿新 • • 發佈：2018-06-21
灰色 file c4c lec complete AC ade ring 裝包 
Hadoop生態圈-Kafka的完全分布式部署
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
本篇博客主要內容就是搭建Kafka完全分布式，它是在kafka本地模式（https://www.cnblogs.com/yinzhengjie/p/9209058.html）的基礎之上進一步實現完全分布式搭建過程。
一.試驗環境
二.kafka完全分布式部署
1>.將kafka加壓後的安裝包發送到其他節點（s102,s103,s104）
[yinzhengjie@s101 data]$ more[yinzhengjie@s101 data]$ more `which xrsync.sh``which xrsync.sh` #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:[email protected] #判斷用戶是否傳參 if [ $# -lt 1 ];then echo "請輸入參數"; exit fi #獲取文件路徑 file=$@ #獲取子路徑 filename=`basename $file` #獲取父路徑 dirpath=`dirname $file` #獲取完整路徑 cd $dirpath fullpath=`pwd -P` #同步文件到DataNodefor (( i=102;i<=105;i++ )) do #使終端變綠色 tput setaf 2 echo =========== s$i %file =========== #使終端變回原來的顏色，即白灰色 tput setaf 7 #遠程執行命令 rsync -lr $filename `whoami`@s$i:$fullpath #判斷命令是否執行成功 if [ $? == 0 ];then echo "命令執行成功" fi done [yinzhengjie@s101 data]$
[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka =========== s102 %file =========== 命令執行成功 =========== s103 %file =========== 命令執行成功 =========== s104 %file =========== 命令執行成功 =========== s105 %file =========== 命令執行成功 [yinzhengjie@s101 data]$[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka
[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka_2.11-1.1.0/ =========== s102 %file =========== 命令執行成功 =========== s103 %file =========== 命令執行成功 =========== s104 %file =========== 命令執行成功 =========== s105 %file =========== 命令執行成功 [yinzhengjie@s101 data]$[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka_2.11-1.1.0/
2>.分發環境變量
[yinzhengjie@s101 data]$ su Password: [root@s101 data]# xrsync.sh /etc/profile =========== s102 %file =========== 命令執行成功 =========== s103 %file =========== 命令執行成功 =========== s104 %file =========== 命令執行成功 =========== s105 %file =========== 命令執行成功 [root@s101 data]# exit exit [yinzhengjie@s101 data]$
3>.修zk節點的改配置文件
[yinzhengjie@s102 ~]$ grep broker.id /soft/kafka/config/server.properties broker.id=102 [yinzhengjie@s102 ~]$ grep listeners /soft/kafka/config/server.properties | grep -v ^# listeners=PLAINTEXT://s102:9092 [yinzhengjie@s102 ~]$修改s102配置文件（/soft/kafka/config/server.properties）
[yinzhengjie@s103 ~]$ grep broker.id /soft/kafka/config/server.properties broker.id=103 [yinzhengjie@s103 ~]$ grep listeners /soft/kafka/config/server.properties | grep -v ^# listeners=PLAINTEXT://s103:9092 [yinzhengjie@s103 ~]$ [yinzhengjie@s103 ~]$修改s103配置文件（/soft/kafka/config/server.properties）
[yinzhengjie@s104 ~]$ grep broker.id /soft/kafka/config/server.properties broker.id=104 [yinzhengjie@s104 ~]$ [yinzhengjie@s104 ~]$ grep listeners /soft/kafka/config/server.properties | grep -v ^# listeners=PLAINTEXT://s104:9092 [yinzhengjie@s104 ~]$修改s104配置文件（/soft/kafka/config/server.properties）
4>.進入zookeeper客戶端並刪除zk的kafka節點數據
[yinzhengjie@s104 ~]$ zkCli.sh Connecting to localhost:2181 2018-06-21 01:23:54,935 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT 2018-06-21 01:23:54,938 [myid:] - INFO [main:Environment@100] - Client environment:host.name=s104 2018-06-21 01:23:54,938 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_131 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/soft/jdk1.8.0_131/jre 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/soft/zk/bin/../build/classes:/soft/zk/bin/../build/lib/*.jar:/soft/zk/bin/../lib/slf4j-log4j12-1.7.25.jar:/soft/zk/bin/../lib/slf4j-api-1.7.25.jar:/soft/zk/bin/../lib/netty-3.10.6.Final.jar:/soft/zk/bin/../lib/log4j-1.2.17.jar:/soft/zk/bin/../lib/jline-0.9.94.jar:/soft/zk/bin/../lib/audience-annotations-0.5.0.jar:/soft/zk/bin/../zookeeper-3.4.12.jar:/soft/zk/bin/../src/java/lib/*.jar:/soft/zk/bin/../conf: 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA> 2018-06-21 01:23:54,940 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux 2018-06-21 01:23:54,941 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64 2018-06-21 01:23:54,941 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-327.el7.x86_64 2018-06-21 01:23:54,941 [myid:] - INFO [main:Environment@100] - Client environment:user.name=yinzhengjie 2018-06-21 01:23:54,941 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/yinzhengjie 2018-06-21 01:23:54,941 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/yinzhengjie 2018-06-21 01:23:54,942 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@277050dc Welcome to ZooKeeper! JLine support is enabled 2018-06-21 01:23:54,973 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2018-06-21 01:23:55,031 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session 2018-06-21 01:23:55,049 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x6800003ae7350004, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] ls / [a, cluster, controller, brokers, zookeeper, yarn-leader-election, hadoop-ha, admin, isr_change_notification, log_dir_event_notification, controller_epoch, consumers, latest_producer_id_block, config, hbase] [zk: localhost:2181(CONNECTED) 1] rmr /controller /brokers /admin /controller_epoch /consumers /latest_producer_id_block /config /isr_change_notification /cluster /log_dir_event_notification [zk: localhost:2181(CONNECTED) 2]
5>.分別啟動s102-s104的kafka
[yinzhengjie@s102 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties [yinzhengjie@s102 ~]$[yinzhengjie@s102 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
[yinzhengjie@s103 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties [yinzhengjie@s103 ~]$[yinzhengjie@s103 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
[yinzhengjie@s104 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties [yinzhengjie@s104 ~]$[yinzhengjie@s104 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
6>.創建主題
[yinzhengjie@s104 ~]$ kafka-topics.sh --zookeeper s102:2181 --list yinzhengjie [yinzhengjie@s104 ~]$查看以及有的主題（[yinzhengjie@s104 ~]$ kafka-topics.sh --zookeeper s102:2181 --list）
[yinzhengjie@s104 ~]$ kafka-topics.sh --zookeeper s104:2181 --create --partitions 2 --replication-factor 1 --topic yzj Created topic "yzj". [yinzhengjie@s104 ~]$創建主題（[yinzhengjie@s104 ~]$ kafka-topics.sh --zookeeper s104:2181 --create --partitions 2 --replication-factor 1 --topic yzj）
7>.在任意zk節點開啟控制臺生產者（例如：在s102上）
[yinzhengjie@s102 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties [yinzhengjie@s102 ~]$ kafka-console-producer.sh --broker-list s104:9092 --topic yzj >尹正傑到此一遊！ >[yinzhengjie@s102 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
8>.在任意zk節點開啟控制臺消費者（例如：在s103上）
[yinzhengjie@s103 ~]$ kafka-console-consumer.sh --zookeeper s102:2181 --topic yzj --from-beginning Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. 尹正傑到此一遊！[yinzhengjie@s103 ~]$ kafka-console-consumer.sh --zookeeper s102:2181 --topic yzj --from-beginning
三.編寫管理腳本
Hadoop生態圈-Kafka的完全分布式部署