kafka1.0.0叢集搭建
一 、kafka安裝配置
1、kafka1.0.0 下載地址
http://kafka.apache.org/downloads.html
2、三臺伺服器名稱
192.168.20.191 master
192.168.20.192 slave1
192.168.20.193 slave2
3、具體安裝配置
1).在主節點192.168.20.191的/usr/tools/kafka路徑下解壓kafka壓縮檔案： tar -zxvf kafka_2.12-1.0.0.tgz
2).檢視配置檔案：/usr/tools/kafka/kafka_2.12-1.0.0/config
3).修改server.properties配置檔案：vim server.properties
mkdir kafka-logs
三臺叢集server.properties配置如下：
192.168.20.191：
broker.id=0
port=9092
host.name=192.168.20.191
log.dirs=/usr/tools/kafka/kafka-logs
zookeeper.connect=192.168.20.191:2181,192.168.20.192:2181,192.168.20.193:2181
192.168.20.192：
broker.id=1
port=9092
host.name=192.168.20.192
log.dirs=/usr/tools/kafka/kafka-logs
zookeeper.connect=192.168.20.191:2181,192.168.20.192:2181,192.168.20.193:2181
192.168.20.193：
broker.id=2
port=9092
host.name=192.168.20.193
log.dirs=/usr/tools/kafka/kafka-logs
zookeeper.connect=192.168.20.191:2181,192.168.20.192:2181,192.168.20.193:2181
4).把整個kafka資料夾遠端複製到slave1和slave2上：
scp -r /usr/tools/kafka
scp -r /usr/tools/kafka [email protected]:/usr/tools/
5).複製過去要記得修改server.properties中的broker.id和host.name
6).叢集啟動和停止（三臺分別執行）：/usr/tools/kafka/kafka_2.12-1.0.0/bin
啟動：
./kafka-server-start.sh -daemon ../config/server.properties &
停止：
./kafka-server-stop.sh
7).配置環境變數(三臺): vim /etc/profile
# KAFKA
export KAFKA_HOME=/usr/tools/kafka/kafka_2.12-1.0.0
export PATH=$PATH:$KAFKA_HOME/bin
使其生效:source /etc/profile
二 、kafka叢集啟動與停止指令碼及其注意事項
1、新建sh指令碼
在 /usr/tools/kafka/kafka_2.12-1.0.0/bin目錄下新建：
touch start-all-kafka.sh
#!/bin/bash
BROKERS="master slave1 slave2"
APPHOME="/usr/tools/kafka/kafka_2.12-1.0.0"
APP_NAME="kafka_2.12-1.0.0"
for i in $BROKERS
do
echo "Starting ${APP_NAME} on ${i} "
ssh ${i} "source /etc/profile; nohup sh ${APPHOME}/bin/kafka-server-start.sh ${APPHOME}/config/server.properties > /dev/null 2>&1 &"
if [[ $? -ne 0 ]]; then
echo "Starting ${APP_NAME} on ${i} is ok"
fi
done
echo All $APP_NAME are started
exit 0
touch stop-all-kafka.sh
#!/bin/bash
BROKERS="master slave1 slave2"
APPHOME="/usr/tools/kafka/kafka_2.12-1.0.0"
APP_NAME="kafka_2.12-1.0.0"
for i in $BROKERS
do
echo "Stopping ${APP_NAME} on ${i} "
ssh ${i} "source /etc/profile;bash ${APPHOME}/bin/kafka-server-stop.sh"
if [[ $? -ne 0 ]]; then
echo "Stopping ${APP_NAME} on ${i} is down"
fi
done
echo All $APP_NAME are stopped
exit 0
2、授權
chmod +x *.sh
3、啟動或停止（./start-all-kafka.sh和./stop-all-kafka.sh）
4、建立和檢視topic（此處建立的是名稱為“my-topic”的topic）
191上建立：
./kafka-topics.sh --create --zookeeper master:2181 --replication-factor 1 --partitions 1 --topic my-topic
191上檢視：
./kafka-topics.sh --list --zookeeper master:2181
5、傳送訊息
191上生產：
./kafka-console-producer.sh --broker-list master:9092 --topic my-topic
6、啟動consumer
192或193上讀取訊息並輸出：
./kafka-console-consumer.sh --zookeeper master:2181 --topic my-topic --from-beginning
