apache kafka原始碼分析工程搭建(IDEA)

阿新 • • 發佈：2019-02-11

[2014-05-24 23:45:31,965] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,009] INFO Property broker.id is overridden to 9 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,009] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2014 
-05-24 23:45:32,009] INFO Property log.dirs is overridden to /Users/lizhitao/kafka-logs (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,009] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property log.retention.hours is overridden to 168 
 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property num.network.threads is overridden to 2 
 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,010] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,011] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,011] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,011] INFO Property zookeeper.connect is overridden to 192.168.2.225:2181,192.168.2.225:2182,192.168.2.225:2183 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,011] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-05-24 23:45:32,032] INFO [Kafka Server 9], starting (kafka.server.KafkaServer)
[2014-05-24 23:45:32,036] INFO [Kafka Server 9], Connecting to zookeeper on 192.168.2.225:2181,192.168.2.225:2182,192.168.2.225:2183 (kafka.server.KafkaServer)
[2014-05-24 23:45:32,045] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-05-24 23:45:32,370] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:host.name=192.168.2.104 (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.version=1.7.0_55 (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.class.path=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/javafx-doclet.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/lizhitao/mt_wp/open_source/kafka-platform/kafka-0.8.1-src/out/production/core:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-library/2.8.0/95bf967bf2e0a26727736228bba3451f4dd3e5b9/scala-library-2.8.0.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/org.apache.zookeeper/zookeeper/3.3.4/6471e17c92181da9e143559c4c4779925a5e6eb0/zookeeper-3.3.4.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/com.101tec/zkclient/0.3/dedcf2b53fb742adba7080ac3aed781694ba616e/zkclient-0.3.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/com.yammer.metrics/metrics-core/2.2.0/f82c035cfa786d3cbec362c38c22a5f5b1bc8724/metrics-core-2.2.0.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/com.yammer.metrics/metrics-annotation/2.2.0/62962b54c490a95c0bb255fa93b0ddd6cc36dd4b/metrics-annotation-2.2.0.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/3.2/d625f12ba08083c8c16dcedd5396ec730e9e77ab/jopt-simple-3.2.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/org.xerial.snappy/snappy-java/1.0.5/10cb4550360a0ec6b80f09a5209d00b6058e82bf/snappy-java-1.0.5.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.15/f0a0d2e29ed910808c33135a3a5a51bba6358f7b/log4j-1.2.15.jar:/Users/lizhitao/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.2/81d61b7f33ebeab314e07de0cc596f8e858d97/slf4j-api-1.7.2.jar:/Applications/IntelliJ IDEA 12.app/lib/idea_rt.jar (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.library.path=/Users/lizhitao/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.io.tmpdir=/var/folders/pn/qjf0v4k52mq965jxjd72hlx00000gp/T/ (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,370] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,371] INFO Client environment:os.version=10.9.2 (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,371] INFO Client environment:user.name=lizhitao (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,371] INFO Client environment:user.home=/Users/lizhitao (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,371] INFO Client environment:user.dir=/Users/lizhitao/mt_wp/open_source/kafka-platform/kafka-0.8.1-src (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,372] INFO Initiating client connection, connectString=192.168.2.225:2181,192.168.2.225:2182,192.168.2.225:2183 sessionTimeout=6000 [email protected] (org.apache.zookeeper.ZooKeeper)
[2014-05-24 23:45:32,387] INFO Opening socket connection to server /192.168.2.225:2181 (org.apache.zookeeper.ClientCnxn)
[2014-05-24 23:45:32,393] ERROR Unable to open socket to 192.168.2.225/192.168.2.225:2181 (org.apache.zookeeper.ClientCnxn)

通過如上7步後就可以正確啟動kafka程式，進行相關debug，並研究其原始碼了。

apache kafka原始碼分析工程搭建(IDEA)

[2014-05-24 23:45:31,965] INFO Verifying properties (kafka.utils.VerifiableProperties) [2014-05-24 23:45:32,009] INFO Property broker.id is overridden to

apache kafka原始碼工程環境搭建(IDEA)

[2014-05-24 23:45:31,965] INFO Verifying properties (kafka.utils.VerifiableProperties) [2014-05-24 23:45:32,009] INFO Property broker.id is overridden to 9

kafka原始碼閱讀環境搭建（gradle構建工具、idea)

1.安裝gradle工具，下載地址：https://gradle.org/next-steps/?version=4.7&format=all2.配置環境變數，GRADLE_HOME,path,注意：要在系統變數中配置3.cmd進入dos視窗，gradle -v檢視版

JDK1.8原始碼分析03之idea搭建原始碼閱讀環境

序言：上一節說了閱讀原始碼的順序，有了一個大體的方向，咱們就知道該如何下手。接下來，就要搭建一個方便閱讀原始碼及debug的環境。有助於跟蹤原始碼的呼叫情況。目前新開發的專案，大多數都是基於JDK1.8開發，所以我選擇該版本進行原始碼分析。 JDK1.8版本號：jdk1.8.0_151 一. JDK1

Kafka 原始碼分析之LogSegment

這裡分析kafka LogSegment原始碼通過一步步分析LogManager,Log原始碼之後就會發現,最終的log操作都在LogSegment上實現.LogSegment負責分片的讀寫恢復重新整理刪除等動作都在這裡實現.LogSegment程式碼同樣在原始碼目錄log下. LogSe

Hadoop原始碼閱讀環境搭建（IDEA） Hadoop原始碼編譯環境搭建 IDEA配置maven中央庫

拿到一份Hadoop原始碼之後，經常關注的兩件事情就是 1、怎麼閱讀？涉及IDEA和Eclipse工程搭建。IDEA搭建，選擇原始碼，逐步匯入即可；Eclipse可以選擇後臺生成工程，也可以選擇IDE匯入。二者工程也可以互相匯入\到處。 2、怎麼構建？利用maven，生成安裝包。二者均需配置maven

[Kafka] Apache Kafka 簡介、叢集搭建及配置詳解

前言 kafka是一種高吞吐量的分散式釋出訂閱訊息系統，它可以處理消費者規模的網站中的所有動作流資料。這種動作（網頁瀏覽，搜尋和其他使用者的行動）是在現代網路上的許多社會功能的一個關鍵因素。這些資料通常是由於吞吐量的要求而通過處理日誌和日誌聚合來解決。 Kafk

Kafka原始碼分析--生產者

ps.本文所有原始碼都基於kafka-0.10.0.1Kafka提供了Java版本的生產者實現--KafkaProducer，使用KafkaProducer的API可以輕鬆實現同步／非同步傳送訊息、批量傳送、超時重發等複雜的功能，KafkaProducer是執行緒安全的，多個

kafka原始碼分析之consumer的原始碼

Consumer的client端示例程式碼 Properties props = new Properties();props.put(ConsumerConfig.BOOTSTRAP_SER

kafka原始碼分析

原文地址：http://www.aboutyun.com/thread-9938-1-1.html 問題導讀1.Kafka提供了Producer類作為java producer的api,此類有幾種傳送方式？2.總結呼叫producer.send方法包含哪些流程？3.Producer難以理解的在什麼地方？ p

kafka原始碼分析之kafkaApis

KafkaApis 說明:用於處理對kafka的訊息請求的中心轉發元件,kafkaapis需要依賴於如下幾個元件: apis = new KafkaApis(socketServer.requestChannel, replicaManager, consumer

kafka原始碼分析之producer

Producer的client端示例程式碼 Properties props = new Properties();props.put("bootstrap.servers", "localhost:9092");props.put("client.id", "De

Kafka原始碼閱讀環境搭建

閒來無事，準備研究一下kafka，分5步走， 1，搭建kafka程式碼閱讀環境 2，搭建kafka執行環境，並測試 3，分析kafka啟動部署原理 4，分析kafka工作機制 5，對關鍵部分進行分析馬上開幹。。。。 1，jdk下載安裝下載地址：連結：http

kafka原始碼分析之一server啟動分析

0. 關鍵概念關鍵概念 ConceptsFunction Topic 用於劃分Message的邏輯概念，一個Topic可以分佈在多個Broker上。 Partition 是Kafka中橫向擴充套件和一切並行化的基礎，每個Topic都至少被切分為1個Partition。

Spark整合Kafka原始碼分析——SparkStreaming從kafak中接收資料

整體概括：要實現SparkStreaming從kafak中接收資料分為以下幾步(其中涉及的類在包org.apache.spark.streaming.kafka中)： 1.建立createStream()函式，返回型別為ReceiverInputDStream物件，在cre

kafka原始碼分析之kafkacluster的管理-KafkaController

KafkaController 說明,這個例項主要用於對kafka cluster進行管理，一個kafka的cluster表示同一個zk環境下所有的broker的集合，在這個cluster中需要有一個broker被選舉成為leader,用於管理其它的broker的上線與

apache kafka原始碼構建打包

準備工作：1.構建kafka的jar並執行打包kafka-0.8.1.1下所有jar，包括core,perf,clients等。[email protected]:~/mt_wp/tmp/ka

Kafka原始碼分析（6）

六其他 1、Producer 首先來看org.apache.kafka.clients.producer中的KafkaProducer類實現，org.apache.kafka開頭的包全部是以java實現的，為使用者提供呼叫kafka元件的功能。該類程式碼如下： pub

Apache Arrow原始碼分析（一）——簡介和框架

背景列儲存在資料庫領域中早已被提出，列儲存資料結構在分析型事務上表現優異，大資料分析引擎，諸如Spark-SQL，Impala 均採用列儲存作為其中間資料表示形式，那麼Apache Arrow就是這樣一種記憶體列式資料結構。在眾多分散式系統中，每

Kafka原始碼分析（4）

四、Replication Subsystem 1、Replica Replica是kafka分發資料的最小單元，主要程式碼如下： class Replica(val brokerId: Int, val partition: Partiti

apache kafka原始碼分析工程搭建(IDEA)

相關推薦