2. 程式人生 > 實用技巧 >快速搭建Zookeeper和Kafka環境

快速搭建Zookeeper和Kafka環境

阿新 • • 發佈：2020-08-19

前言

由於專案需要涉及到zookeeper和Kafka的使用，快速做了一篇筆記，方便小夥伴們搭建環境。

zookeeper

官方定義

What is ZooKeeper?

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper是一個分散式的，開放原始碼的分散式應用程式協調服務，是Google的Chubby一個開源的實現，是Hadoop和Hbase的重要元件。它是一個為分散式應用提供一致性服務的軟體，提供的功能包括：配置維護、域名服務、分散式同步、組服務等。
ZooKeeper的目標就是封裝好複雜易出錯的關鍵服務，將簡單易用的介面和效能高效、功能穩定的系統提供給使用者。

zookeeper常用埠號：2181

環境搭建

  1. 下載zookeeper：

    下載地址：http://mirrors.hust.edu.cn/apache/zookeeper/

  2. 在zookeeper-3.4.10目錄下，新建資料夾，並命名，例如data。(路徑為：D:\zookeeper-3.4.10\conf\data)

  3. 複製“zoo_sample.cfg”副本à並將副本重新命名為“zoo.cfg，在任意文字編輯器（eg：記事本）中開啟zoo.cfg，找到並編輯dataDir=D:\\zookeeper-3.4.10\\data

  4. windows的話，在cmd下執行：D:\zookeeper-3.4.13\bin>zkServer.cmd

D:\zookeeper-3.4.13\bin>zkServer.cmd

D:\zookeeper-3.4.13\bin>call "C:\Program Files\Java\jdk1.8.0_181"\bin\java "-Dzookeeper.log.dir=D:\zookeeper-3.4.13\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "D:\zookeeper-3.4.13\bin\..\build\classes;D:\zookeeper-3.4.13\bin\..\build\lib\*;D:\zookeeper-3.4.13\bin\..\*;D:\zookeeper-3.4.13\bin\..\lib\*;D:\zookeeper-3.4.13\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg"
2019-03-18 22:07:46,267 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg
2019-03-18 22:07:46,276 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-03-18 22:07:46,276 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-03-18 22:07:46,277 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-03-18 22:07:46,280 [myid:] - WARN  [main:QuorumPeerMain@116] - Either no config or no quorum defined in config, running  in standalone mode
2019-03-18 22:07:46,397 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg
2019-03-18 22:07:46,398 [myid:] - INFO  [main:ZooKeeperServerMain@98] - Starting server
2019-03-18 22:07:46,422 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2019-03-18 22:07:46,422 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=DESKTOP-D5MA1P9
2019-03-18 22:07:46,422 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.8.0_181
2019-03-18 22:07:46,423 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2019-03-18 22:07:46,423 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre
2019-03-18 22:07:46,423 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=D:\zookeeper-3.4.13\bin\..\build\classes;D:\zookeeper-3.4.13\bin\..\build\lib\*;D:\zookeeper-3.4.13\bin\..\zookeeper-3.4.13.jar;D:\zookeeper-3.4.13\bin\..\lib\audience-annotations-0.5.0.jar;D:\zookeeper-3.4.13\bin\..\lib\jline-0.9.94.jar;D:\zookeeper-3.4.13\bin\..\lib\log4j-1.2.17.jar;D:\zookeeper-3.4.13\bin\..\lib\netty-3.10.6.Final.jar;D:\zookeeper-3.4.13\bin\..\lib\slf4j-api-1.7.25.jar;D:\zookeeper-3.4.13\bin\..\lib\slf4j-log4j12-1.7.25.jar;D:\zookeeper-3.4.13\bin\..\conf
2019-03-18 22:07:46,424 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Java\jre1.8.0_191\bin;C:\Program Files\Java\jdk1.8.0_181\bin;C:\Users\yzd\AppData\Local\Microsoft\WindowsApps;C:\Users\yzd\AppData\Roaming\npm;.
2019-03-18 22:07:46,425 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=C:\Users\yzd\AppData\Local\Temp\
2019-03-18 22:07:46,425 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
2019-03-18 22:07:46,427 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Windows 10
2019-03-18 22:07:46,427 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2019-03-18 22:07:46,427 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=10.0
2019-03-18 22:07:46,428 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=yzd
2019-03-18 22:07:46,428 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=C:\Users\yzd
2019-03-18 22:07:46,428 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=D:\zookeeper-3.4.13\bin
2019-03-18 22:07:46,438 [myid:] - INFO  [main:ZooKeeperServer@836] - tickTime set to 2000
2019-03-18 22:07:46,438 [myid:] - INFO  [main:ZooKeeperServer@845] - minSessionTimeout set to -1
2019-03-18 22:07:46,439 [myid:] - INFO  [main:ZooKeeperServer@854] - maxSessionTimeout set to -1
2019-03-18 22:07:46,647 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-03-18 22:07:46,651 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181

搭建成功

zookeeper和eureka比較

ZooKeeper、Eureka對比：

https://www.cnblogs.com/jieqing/p/8394001.html

CAP定理（C-資料一致性；A-服務可用性；P-服務對網路分割槽故障的容錯性，這三個特性在任何分散式系統中不能同時滿足，最多同時滿足兩個）

ZooKeeper基於CP，不保證高可用，如果zookeeper正在選主，或者Zookeeper叢集中半數以上機器不可用，那麼將無法獲得資料。Eureka基於AP，能保證高可用，即使所有機器都掛了，也能拿到本地快取的資料。作為註冊中心，其實配置是不經常變動的，只有發版和機器出故障時會變。對於不經常變動的配置來說，CP是不合適的，而AP在遇到問題時可以用犧牲一致性來保證可用性，既返回舊資料，快取資料。

所以理論上Eureka是更適合作註冊中心。而現實環境中大部分專案可能會使用ZooKeeper，那是因為叢集不夠大，並且基本不會遇到用做註冊中心的機器一半以上都掛了的情況。所以實際上也沒什麼大問題。

kfaka

Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

Kafka是由Apache軟體基金會開發的一個開源流處理平臺，由Scala和Java編寫。Kafka是一種高吞吐量的分散式釋出訂閱訊息系統，它可以處理消費者規模的網站中的所有動作流資料。 這種動作（網頁瀏覽，搜尋和其他使用者的行動）是在現代網路上的許多社會功能的一個關鍵因素。 這些資料通常是由於吞吐量的要求而通過處理日誌和日誌聚合來解決。 對於像Hadoop一樣的日誌資料和離線分析系統，但又要求實時處理的限制，這是一個可行的解決方案。Kafka的目的是通過Hadoop的並行載入機制來統一線上和離線的訊息處理，也是為了通過叢集來提供實時的訊息。

kafka包內自帶整合好的zookeeper，所以如果要搭建kafka可以省去單獨下載zookeeper的步驟。想起了欲練武功必先自宮的笑話，哈哈哈。

環境搭建

  1. 下載地址：http://kafka.apache.org/downloads.html
  2. 解壓縮，建立一個空資料夾 logs. eg: D:\kafka_2.11-1.0.0\logs
  3. 進入config目錄，編輯 server.properties檔案(eg: 用“寫字板”開啟)。找到並編輯log.dirs= D:\kafka_2.11-1.0.0\logs
  4. 請確保在啟動Kafka伺服器前，Zookeeper例項已經準備好並開始執行。
  5. 在此處開啟Powershell視窗，執行：.\bin\windows\kafka-server-start.bat .\config\server.properties

PS D:\kafka_2.12-2.1.1> .\bin\windows\kafka-server-start.bat .\config\server.properties
[2019-03-18 22:15:34,866] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-03-18 22:15:35,503] INFO starting (kafka.server.KafkaServer)
[2019-03-18 22:15:35,505] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-03-18 22:15:35,530] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,536] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,536] INFO Client environment:host.name=DESKTOP-D5MA1P9 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,536] INFO Client environment:java.version=1.8.0_181 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.class.path=D:\kafka_2.12-2.1.1\libs\activation-1.1.1.jar;D:\kafka_2.12-2.1.1\libs\aopalliance-repackaged-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\argparse4j-0.7.0.jar;D:\kafka_2.12-2.1.1\libs\audience-annotations-0.5.0.jar;D:\kafka_2.12-2.1.1\libs\commons-lang3-3.8.1.jar;D:\kafka_2.12-2.1.1\libs\connect-api-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-basic-auth-extension-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-file-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-json-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-runtime-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-transforms-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\guava-20.0.jar;D:\kafka_2.12-2.1.1\libs\hk2-api-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\hk2-locator-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\hk2-utils-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\jackson-annotations-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-core-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-databind-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-jaxrs-base-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-jaxrs-json-provider-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-module-jaxb-annotations-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\javassist-3.22.0-CR2.jar;D:\kafka_2.12-2.1.1\libs\javax.annotation-api-1.2.jar;D:\kafka_2.12-2.1.1\libs\javax.inject-1.jar;D:\kafka_2.12-2.1.1\libs\javax.inject-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\javax.servlet-api-3.1.0.jar;D:\kafka_2.12-2.1.1\libs\javax.ws.rs-api-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\javax.ws.rs-api-2.1.jar;D:\kafka_2.12-2.1.1\libs\jaxb-api-2.3.0.jar;D:\kafka_2.12-2.1.1\libs\jersey-client-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-common-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-container-servlet-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-container-servlet-core-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-hk2-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-media-jaxb-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-server-2.27.jar;D:\kafka_2.12-2.1.1\libs\jetty-client-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-continuation-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-http-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-io-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-security-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-server-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-servlet-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-servlets-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-util-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jopt-simple-5.0.4.jar;D:\kafka_2.12-2.1.1\libs\kafka-clients-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-log4j-appender-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-examples-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-scala_2.12-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-test-utils-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-tools-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-javadoc.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-javadoc.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-scaladoc.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-scaladoc.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-sources.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-sources.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test-sources.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test-sources.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1.jar.asc;D:\kafka_2.12-2.1.1\libs\log4j-1.2.17.jar;D:\kafka_2.12-2.1.1\libs\lz4-java-1.5.0.jar;D:\kafka_2.12-2.1.1\libs\maven-artifact-3.6.0.jar;D:\kafka_2.12-2.1.1\libs\metrics-core-2.2.0.jar;D:\kafka_2.12-2.1.1\libs\osgi-resource-locator-1.0.1.jar;D:\kafka_2.12-2.1.1\libs\plexus-utils-3.1.0.jar;D:\kafka_2.12-2.1.1\libs\reflections-0.9.11.jar;D:\kafka_2.12-2.1.1\libs\rocksdbjni-5.14.2.jar;D:\kafka_2.12-2.1.1\libs\scala-library-2.12.7.jar;D:\kafka_2.12-2.1.1\libs\scala-logging_2.12-3.9.0.jar;D:\kafka_2.12-2.1.1\libs\scala-reflect-2.12.7.jar;D:\kafka_2.12-2.1.1\libs\slf4j-api-1.7.25.jar;D:\kafka_2.12-2.1.1\libs\slf4j-log4j12-1.7.25.jar;D:\kafka_2.12-2.1.1\libs\snappy-java-1.1.7.2.jar;D:\kafka_2.12-2.1.1\libs\validation-api-1.1.0.Final.jar;D:\kafka_2.12-2.1.1\libs\zkclient-0.11.jar;D:\kafka_2.12-2.1.1\libs\zookeeper-3.4.13.jar;D:\kafka_2.12-2.1.1\libs\zstd-jni-1.3.7-1.jar (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,538] INFO Client environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Java\jre1.8.0_191\bin;C:\Program Files\Java\jdk1.8.0_181\bin;C:\Users\yzd\AppData\Local\Microsoft\WindowsApps;C:\Users\yzd\AppData\Roaming\npm;. (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,539] INFO Client environment:java.io.tmpdir=C:\Users\yzd\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,542] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,542] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,543] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,543] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.name=yzd (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.home=C:\Users\yzd (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.dir=D:\kafka_2.12-2.1.1 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,549] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3e2e18f2 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,570] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,570] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,574] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,598] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1000ae3e3690000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,602] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,974] INFO Cluster ID = brvhJ2K3R4qZnGJ6CQhghA (kafka.server.KafkaServer)
[2019-03-18 22:15:35,980] WARN No meta.properties file under dir D:\kafka_2.12-2.1.1\logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-03-18 22:15:36,049] INFO KafkaConfig values:
        advertised.host.name = null
        ...(省略若干)
        zookeeper.sync.time.ms = 2000
 (kafka.server.KafkaConfig)
[2019-03-18 22:15:36,092] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,092] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,094] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,132] INFO Loading logs. (kafka.log.LogManager)
[2019-03-18 22:15:36,143] INFO Logs loading complete in 11 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,161] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,166] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,486] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2019-03-18 22:15:36,534] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2019-03-18 22:15:36,562] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,563] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,563] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,581] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-03-18 22:15:36,607] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,612] INFO Result of znode creation at /brokers/ids/0 is: OK (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,613] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(DESKTOP-D5MA1P9,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,615] WARN No meta.properties file under dir D:\kafka_2.12-2.1.1\logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-03-18 22:15:36,682] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,684] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,685] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,689] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,714] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2019-03-18 22:15:36,716] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2019-03-18 22:15:36,731] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 15 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-18 22:15:36,748] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2019-03-18 22:15:36,780] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-03-18 22:15:36,782] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-03-18 22:15:36,791] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-03-18 22:15:36,875] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-03-18 22:15:36,892] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2019-03-18 22:15:36,901] INFO Kafka version : 2.1.1 (org.apache.kafka.common.utils.AppInfoParser)
[2019-03-18 22:15:36,919] INFO Kafka commitId : 21234bee31165527 (org.apache.kafka.common.utils.AppInfoParser)
[2019-03-18 22:15:36,922] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

搭建成功。

測試kafka生產者消費者

  1. 開啟zookeeper和kafka服務

  2. 建立topic：

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

看一眼剛才建立的主題

> bin/kafka-topics.sh --list --zookeeper localhost:2181
test
  1. 發訊息：接收者
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
test

  1. 收訊息：消費者
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message

參考

https://blog.csdn.net/tianmanchn/article/details/78943147

https://www.jianshu.com/p/ca60bc8c5673

關注我

我是蠻三刀把刀，碩士，後端開發工程師。主要關注後端開發，資料安全，爬蟲等方向。微信：yangzd1102

Github個人主頁：

https://github.com/qqxx6661

原創部落格主要內容

  • Java知識點複習全手冊
  • Leetcode演算法題解析
  • 劍指offer演算法題解析
  • Python爬蟲相關技術實戰
  • 後端開發相關技術實戰
  • SpringCloud實戰

個人公眾號：後端技術漫談

如果文章對你有幫助，不妨收藏起來並轉發給您的朋友們~

相關推薦

快速搭建Zookeeper和Kafka環境

前言 由於專案需要涉及到zookeeper和Kafka的使用，快速做了一篇筆記，方便小夥伴們搭建環境。

快速搭建kubernetes與kubeSphere環境（親測有效）

最近在學習k8s，但是安裝k8s實在太麻煩了，也耗費了不少時間，下面來說說安裝步驟。以下是我的機器配置，其實2核4G也可以，但是因為我想學習下kubeSphere就要求配置高了，我在這上面也耗費了不少時間，鄙人特將學習過

Docker搭建Zookeeper&Kafka叢集

前排提示：最新的docker-compole.yml請去github獲取，README有相應的操作步驟。 Github地址：github.com/JacianLiu/d…

04_Linux搭建Jdk和tomcat環境

由於redhat自帶安裝了jdk，我們需要現將其進行解除安裝 1、 先檢查自帶的JDK rpm -qa|grep gcj

windows下一鍵快速搭建PHP本地執行環境

技術標籤：php windows下一鍵快速搭建PHP本地執行環境的好工具(含php7.0環境) 推薦幾款一鍵快速搭建PHP本地執行環境的好工具(含php7.0及apache，nigix，mysql) 首推phpstudy2016和wampServer3.0.6 理由支援php7.

docker（3）快速搭建centos7-python3.6環境

前言 當我們在一臺電腦上搭建了python3.6的環境，下次換臺電腦，又得重新搭建一次，設定環境變數等操作。

使用Docker快速搭建ZooKeeper叢集

前言 之前，已經使用 Docker 安裝 zookeeper，但是這只是安裝單機的。現在，我們要用 docker-compose 來編排叢集容器。

阿里雲-快速搭建Java Web開發環境

2 2. 安裝JDK 1. 執行以下命令，檢視yum源中JDK版本。 yum list java* 2. 執行以下命令，使用yum安裝JDK1.8。

centos安裝ZooKeeper和kafka

A Zookeeper安裝與配置一：下載zookeeper安裝包從官網下載：http://mirrors.hust.edu.cn/apache/zookeeper/二：上傳並配置zookeeper1.放在/usr/local目錄下並解壓。tar -zxvf zookeeper-3.4.13.tar.gz即zookeeper路徑

k8s下部署zookeeper和kafka叢集

zookeeper配置istio sidecar後存在的網路不可用問題 如果zookeeper配置了istio sidecar ，在選舉階段就會報connection refused（Connection refused）錯誤

快速搭建 SpringCloud 微服務開發環境的腳手架

本文作者：HelloGitHub-秦人 本文適合有 SpringBoot 和 SpringCloud 基礎知識的人群，跟著本文可使用和快速搭建 SpringCloud 專案。

IDEA快速搭建Java開發環境的教程圖解

作為IntelliJ IDEA mac新手，IDEA如何快速搭建Java開發環境呢？ 今天小編就給大家帶來了IntelliJ IDEA mac使用教程，想知道IDEA如何快速搭建Java開發環境？那就一起來看看吧！

python安裝和pycharm環境搭建設定方法

一、python 開發工具的選擇 python開發工具有很多，這裡推薦使用 pycharm；因為jetbrains公司擁有眾多的開發工具以及開發擁護者。這些工具有相同的風格和各類使用案例、描述問題的部落格，可供初學者參考。

Window7搭建Kafka環境總結

1.安裝zooeleeper 下載連結：http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/ 安裝步驟如下：

如何快速搭建一個自己的伺服器的詳細教程（java環境）

一.伺服器的購買 1. 我選擇的是阿里雲的伺服器，學生價9.5元一個月，百度直接搜尋阿里雲，然後點選右上角登入，推薦大家用支付寶掃碼登入，方便快捷。阿里雲官網的東西比較多，登入後我找了很久也沒有找到學生伺服器

Mesos+Zookeeper+Marathon+Docker環境搭建

相關理論請參考：https://www.cnblogs.com/Bourbon-tian/p/7155054.html，本文基於https://www.cnblogs.com/Bourbon-tian/p/7151840.html（文章時2017年寫的）進行了實踐，針對新版mesos部署會出現的問題提供瞭解決方

手把手教你搭建zookeeper+kafka叢集

1. kafka的定義 　　kafka是一個分散式訊息系統，由linkedin使用scala編寫，用作LinkedIn的活動流（Activity Stream）和運營資料處理管道（Pipeline）的基礎。具有高水平擴充套件和高吞吐量。

kafka-環境搭建與簡單入門

Kafka環境搭建與簡單入門 步驟一：下載程式碼 下載Kafka安裝包，並解壓。 這裡下載的版本為kafka_2.11-1.0.0.tgz。

快速搭建Docker環境

本教程介紹如何快速搭建Docker環境，並使用Docker部署一個Nginx服務。 體驗目標

docker快速搭建禪道環境

前言 bug管理系統是每個公司測試團隊必備的，當去一個新的公司組建一個測試團隊的時候，需快速搭建一套bug管理系統，使用docker搭建禪道就非常方便。