wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
[[email protected] ~]# tar -zxvf zookeeper-3.4.13.tar.gz -C /usr/local/
[[email protected] ~]# vi /etc/profile
增加下列兩行
export ZK_HOME=/usr/local/zookeeper-3.4.13
export PATH=$ZK_HOME/bin:$PATH
[[email protected]
 
 ~]# source /etc/profile
[[email protected] ~]# cd /usr/local/zookeeper-3.4.13/
[[email protected] zookeeper-3.4.13]# cd conf/
[[email protected] conf]# mv zoo_sample.cfg zoo.cfg
[[email protected] conf]# cat /etc/hosts
192.168.229.133 testceph
192.168.229.130 redis1
192.168.229.131 redis2
[[email protected]
 
 conf]# vi zoo.cfg
dataDir = /usr/local/zookeeper-3.4.13/data
server.0=192.168.229.133:2888:3888
server.1=192.168.229.130:2888:3888
server.2=192.168.229.131:2888:3888

server.X=A:B:C
X-代表伺服器編號
A-代表ip
B和C-代表埠，這個埠用來系統之間通訊
[[email protected] conf]# cd ../
[[email protected] zookeeper-3.4.13]# mkdir data
[
 
[email protected] zookeeper-3.4.13]# cd data/
[[email protected] data]# vi myid
0

#之後會產生一個新檔案，直接在裡面寫 X 即可
#比如我配置的三個server，myid裡面寫的X就是server.X=ip:2888:3888 中ip所對應的X
 server.0=192.168.229.133:2888:3888【192.168.229.133伺服器上面的myid填寫0】
 server.1=192.168.229.130:2888:3888【192.168.229.130伺服器上面的myid填寫1】
 server.2=192.168.229.131:2888:3888【192.168.229.131伺服器上面的myid填寫2】

把原始碼包和配置檔案考入到其它機器中
[[email protected] local]# scp /etc/profile [email protected]:/etc/profile
[[email protected] local]# scp /etc/profile [email protected]:/etc/profile
[[email protected] local]# scp -r zookeeper-3.4.13 [email protected]:/usr/local/
[[email protected] local]# scp -r zookeeper-3.4.13 [email protected]:/usr/local/

修改所有機器中的vi /usr/local/zookeeper-3.4.13/data/myid裡面的資料，0，1，2
確保與server.0, server.1, server.2 對應起來

啟動zookeeper.sh
[[email protected] ~]# zkServer.sh start
[[email protected] ~]# zkServer.sh start
[[email protected] ~]# zkServer.sh start

是否啟動成功，執行一下命令
[[email protected] ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: follower
[[email protected] ~]# ssh redis1
Last login: Sun Dec 23 01:32:26 2018 from testceph
[[email protected] ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: follower
[[email protected] ~]#
[[email protected] ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: leader
[[email protected] ~]#

檢視啟動程序
[[email protected] ~]# zkServer.sh start-foreground

測試[[email protected] ~]# zkCli.sh -server testceph:2181



下載kafka
http://kafka.apache.org/downloads
[[email protected] ~]# wget http://mirror.bit.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz
[[email protected] ~]# tar -zxvf kafka_2.12-2.1.0.tgz -C /usr/local/
[[email protected] local]# ln -s kafka_2.12-2.1.0 kafka
[[email protected] local]# cd kafka
[[email protected] kafka]# vi config/server.properties
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.partitions=1   #然後下面num.partitions是預設單個broker上的partitions數量，預設是1個，如果想提高單機的併發效能，這裡可以配置多個
log.retention.hours=168 #然後是kafka日誌的保留時間，單位小時，預設是168小時，也就是7天
zookeeper.connect=testceph:2181,redis1:2181,redis2:2181 #zookeeper叢集
zookeeper.connection.timeout.ms=6000  #超時時間
[[email protected] local]# scp -r kafka_2.12-2.1.0 [email protected]:/usr/local/
[[email protected] local]# scp -r kafka_2.12-2.1.0 [email protected]:/usr/local/
[[email protected] local]# vi /etc/profile
export kafka_HOME=/usr/local/kafka_2.12-2.1.0
export PATH=$kafka_HOME/bin:$PATH

[[email protected] ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.12-2.1.0/config/server.properties


kafka壓力測試
1建立topic
[[email protected] ~]# kafka-topics.sh --zookeeper 192.168.229.133:2181 --create --topic test --partitions 3 --replication-factor 1
Created topic "test".
[[email protected] ~]#

刪除topic
[[email protected] ~]# kafka-topics.sh --delete --zookeeper testceph:2181 --topic test

檢視zookeeper和kafka是否啟動成功

一 建立topic
使用 replica-assignment 引數手動指定 Topic Partition Replica 與 Kafka Broker 之間的儲存對映關係。
[[email protected] ~]# kafka-topics.sh --create --zookeeper testceph:2181,redis1:2181,redis2:2181 --topic topic_name --replica-assignment 0:1,1:2,2:0
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_name".

0:1,1:2,2:0中的數字均為 broker.id；3個分割槽(逗號分隔)；每個分割槽有兩個副本(副本所在的 broker 以冒號分割)

使用 partitions 和 replication-factor 引數自動分配儲存對映關係。

[[email protected] ~]# kafka-topics.sh --create --zookeeper testceph:2181,redis1:2181,redis2:2181 --topic topic_name1 --partitions 3 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_name1".
[[email protected] ~]#


建立topic過程的問題，replication-factor個數不能超過broker的個數，否則有如下錯誤資訊：
[[email protected] ~]# kafka-topics.sh --create --zookeeper testceph:2181,redis1:2181,redis2:2181 --topic topic_name2 --partitions 3 --replication-factor 2 --config cleanup.policy=compact --config retention.ms=500
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Replication factor: 2 larger than available brokers: 1.
[2018-12-23 04:16:11,404] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.
 (kafka.admin.TopicCommand$)


二檢視topic
2.1檢視topic列表，查詢出來的結果僅有topic的名稱資訊
[[email protected] ~]# kafka-topics.sh --list --zookeeper testceph:2181,redis1:2181,redis2:2181
topic_name
topic_name1
[[email protected] ~]#

2.2檢視指定topic明細
[[email protected] ~]# kafka-topics.sh --describe --zookeeper testceph:2181,redis1:2181,redis2:2181 --topic topic_name
Topic:topic_name   PartitionCount:3   ReplicationFactor:2    Configs:
   Topic: topic_name  Partition: 0   Leader: 0  Replicas: 0,1  Isr: 0
   Topic: topic_name  Partition: 1   Leader: none   Replicas: 1,2  Isr:
   Topic: topic_name  Partition: 2   Leader: 0  Replicas: 2,0  Isr: 0
[[email protected] ~]#
PartitionCount: partition個數
ReplicationFactor:副本個數
Partition: partition編號，從0開始遞增
Leader:但前partition起作用的breaker.id
Replicas:當前副本資料所在的breaker.id,是一個列表，排在最前面的起作用
lsr:當前kakfa叢集中可用的breaker.id列表

三刪除topic
[[email protected] ~]# kafka-topics.sh --delete --zookeeper testceph:2181,redis1:2181,redis2:2181 --topic topic_name
Topic topic_name is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[[email protected] ~]#
若 delete.topic.enable=true
直接徹底刪除該 Topic。
若 delete.topic.enable=false
如果當前 Topic 沒有使用過即沒有傳輸過資訊：可以徹底刪除。
如果當前 Topic 有使用過即有過傳輸過資訊：並沒有真正刪除 Topic 只是把這個 Topic 標記為刪除(marked for deletion)，重啟 Kafka Server 後刪除。
注：delete.topic.enable=true 配置資訊位於配置檔案 config/server.properties 中(較新的版本中無顯式配置，預設為 true)。

四修改topic