一、使用技術版本

kafka_2.10-0.10.2.1.tar 

zookeeper-3.4.5.tar

二、環境搭建

因為kafka環境依賴於zookeeper，所以先搭建zookeeper

1、zookeeper搭建

先建立一個資料夾，存放壓縮包，並解壓

[email protected]:~# cd /wingcloud
[email protected]:/wingcloud# ls
kafka_2.10-0.10.2.1.tar  zookeeper-3.4.5.tar
[email protected]:/wingcloud# tar -xvf zookeeper-3.4.5.tar
 

解壓後，將資料夾放到/usr/local下

[email protected]:/wingcloud# mv zookeeper-3.4.5 /usr/local/zk

進入/usr/local/zk/conf執行如下

[email protected]:/wingcloud# cp zoo_sample.cfg zo.cfg
[email protected]:/wingcloud# mv zo.cfg zoo.cfg
[email protected]:/wingcloud# vim zoo.cfg

進入zoo.cfg後只需修改dataDir

修改如下,儲存退出。

dataDir=/usr/local/zk/data/

返回到zk目錄下建立data目錄

[email protected]:/usr/local/zk# mkdir data

新增zk的環境變數

[email protected]:/usr/local/zk# cd ..
[email protected]:/usr/local# vim /etc/profile
[email protected]:/usr/local# ls
bin                  etc    include  logstash  qcloud  share  yd.socket.server
elasticsearch-2.4.6  games  lib      man       sbin    src    zk
 
[email protected]:/usr/local# vim /etc/profile

ZK_HOME=/usr/local/zk
PATH=$ZK_HOME/bin:$PATH

[email protected]:/usr/local# source /etc/profile

最後啟動zk，並檢視其是否成功啟動

[email protected]:/usr/local# zkServer.sh start
JMX enabled by default
Using config: /usr/local/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[email protected]:/usr/local# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zk/bin/../conf/zoo.cfg
Mode: standalone

2、kafka搭建

解壓並將檔案轉移到/usr/local下

[email protected]:/wingcloud# tar -xvf kafka_2.10-0.10.2.1.tar
[email protected]:/wingcloud# ls
kafka_2.10-0.10.2.1  kafka_2.10-0.10.2.1.tar  zookeeper-3.4.5.tar
[email protected]:/wingcloud# mv kafka_2.10-0.10.2.1 /usr/local
[email protected]:/wingcloud# cd /usr/local
[email protected]:/usr/local# ls
bin                  include              man     src
elasticsearch-2.4.6  kafka_2.10-0.10.2.1  qcloud  yd.socket.server
etc                  lib                  sbin    zk
games                logstash             share   zookeeper.out

修改server.properties

[email protected]:/usr/local# cd kafka_2.10-0.10.2.1
[email protected]:/usr/local/kafka_2.10-0.10.2.1# ls
bin  config  libs  LICENSE  NOTICE  site-docs
[email protected]:/usr/local/kafka_2.10-0.10.2.1# cd config
[email protected]:/usr/local/kafka_2.10-0.10.2.1/config# ls
connect-console-sink.properties    consumer.properties
connect-console-source.properties  log4j.properties
connect-distributed.properties     producer.properties
connect-file-sink.properties       server.properties
connect-file-source.properties     tools-log4j.properties
connect-log4j.properties           zookeeper.properties
connect-standalone.properties
[email protected]:/usr/local/kafka_2.10-0.10.2.1/config# vim server.properties

在server.properties中找到Log Basics註釋，修改其下面的log.dirs如下

log.dirs=/usr/local/kafka_2.10-0.10.2.1/data/kafka-logs

其中也可以修改zk相關，因為是本機這邊就不用配了，ok，儲存退出

啟動kafka

[email protected]:/usr/local/kafka_2.10-0.10.2.1/config# cd ..
[email protected]:/usr/local/kafka_2.10-0.10.2.1# ls
bin  config  libs  LICENSE  NOTICE  site-docs
[email protected]:/usr/local/kafka_2.10-0.10.2.1# cd bin
[email protected]:/usr/local/kafka_2.10-0.10.2.1/bin# ls
connect-distributed.sh               kafka-replica-verification.sh
connect-standalone.sh                kafka-run-class.sh
kafka-acls.sh                        kafka-server-start.sh
kafka-broker-api-versions.sh         kafka-server-stop.sh
kafka-configs.sh                     kafka-simple-consumer-shell.sh
kafka-console-consumer.sh            kafka-streams-application-reset.sh
kafka-console-producer.sh            kafka-topics.sh
kafka-consumer-groups.sh             kafka-verifiable-consumer.sh
kafka-consumer-offset-checker.sh     kafka-verifiable-producer.sh
kafka-consumer-perf-test.sh          windows
kafka-mirror-maker.sh                zookeeper-security-migration.sh
kafka-preferred-replica-election.sh  zookeeper-server-start.sh
kafka-producer-perf-test.sh          zookeeper-server-stop.sh
kafka-reassign-partitions.sh         zookeeper-shell.sh
kafka-replay-log-producer.sh
[email protected]:/usr/local/kafka_2.10-0.10.2.1/bin# ./kafka-server-start.sh  ../config/server.properties

啟動完成後，就算搭建完成了，接下來測試一下

需要另外再開一個終端，進入linux伺服器

在新開的終端下需要建立topics，先進入kafka/bin下

解釋下命令：--zookeeper 127.0.0.1:2181指定zk。 --partitions 1分割槽設為1個，因為是單機。--replication-factor 1副本設為1個。--topic wingcloud設定topic名字叫wingcloud

[email protected]:/usr/local/kafka_2.10-0.10.2.1/bin$ ./kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --partitions 1  --replication-factor 1 --topic wingcloud
Created topic "wingcloud".

然後使用生產者，--broker-list 127.0.0.1:9092指kafka服務，叢集時寫多個即可

[email protected]:/usr/local/kafka_2.10-0.10.2.1/bin$ ./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic wingcloud

啟動後，再開一個終端

在新開的終端下使用消費者

[email protected]:~$ cd /usr/local
[email protected]:/usr/local$ ls
bin                  include              man     src
elasticsearch-2.4.6  kafka_2.10-0.10.2.1  qcloud  yd.socket.server
etc                  lib                  sbin    zk
games                logstash             share   zookeeper.out
[email protected]:/usr/local$ cd kafka_2.10-0.10.2.1
[email protected]:/usr/local/kafka_2.10-0.10.2.1$ ls
bin  config  data  libs  LICENSE  logs  NOTICE  site-docs
[email protected]:/usr/local/kafka_2.10-0.10.2.1$ cd bin
[email protected]:/usr/local/kafka_2.10-0.10.2.1/bin$ ./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic wingcloud

消費者啟動之後，進行資料測試。

在之前的生產者終端輸入資料，在消費者終端也能顯示的話，測試成功，如下

好了kafka搭建完成