Kafka的使用和錯誤解決

一、下載kafka解壓縮：配置環境變數

vim /etc/profile

export KAFKA_HOME=/root/kafka_2.11-1.0.0
export PATH=$PATH:$KAFKA_HOME/bin

source /etc/profile

二 、kafka中需要使用zookeeper

1. 先將zookeeper啟動,如果在偽分散式下,kafka已經集成了zk，在kafka中的config目錄下。

      可以編輯config/zookeeper.properties修改zookeeper的埠號。

   後臺啟動zookeeper：
   [[email protected]

   bin]# nohup zookeeper-server-start.sh ../config/zookeeper.properties &

2. 啟動broker

`[[email protected] bin]# nohup kafka-server-start.sh ../config/server.properties &`

3.測試:模擬訊息的消費和生產

（1）建立主題

[[email protected] bin]# kafka-topics.sh --create --zookeeper localhost:2281 --topic KafkaTestTopic --partitions 1 --replication-factor 1

Created topic "KafkaTestTopic".
 

（2）建立生產者

[[email protected] bin]# kafka-console-producer.sh --topic KafkaTestTopic --broker-list localhost:9092

檢視server.properties中的#listeners=PLAINTEXT://:9092，獲取kafka的埠

（3）建立消費者

[[email protected] bin]# kafka-console-consumer.sh --topic KafkaTestTopic --zookeeper localhost:2281

三、可能出現的錯誤：

（1）
[[email protected] bin]# kafka-topics.sh --create --zookeeper localhost:2281 --topic KafkaTestTopic --partitions 1 --replication-factor 1
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2018-11-20 16:44:16,269] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
(kafka.admin.TopicCommand$)

解決：修改server.properties中的：zookeeper.connect=localhost:2281，讓2281埠號和zookeeper.properties中的zookeeper埠號一致，然後重啟kafka。**

（2）
kafka.common.KafkaException: fetching topic metadata for topics [Set(KafkaTestTopic)] from broker [ArrayBuffer(BrokerEndPoint(0,123.125.50.7,9092))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
（3）
[2018-11-20 17:28:53,411] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 52 : {KafkaTestTopic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-11-20 17:28:53,513] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 53 : {KafkaTestTopic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-11-20 17:28:53,617] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 54 : {KafkaTestTopic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-11-20 17:28:53,721] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 55 : {KafkaTestTopic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

解決(2)和（3)的錯誤:
修改server.properties中的
(1) listeners=PLAINTEXT://localhost:9092
(2) advertised.listeners=PLAINTEXT://localhost:9092