Kafka是由LinkedIn開發的一個分布式的消息系統，使用Scala編寫，它以可水平擴展和高吞吐率而被廣泛使用。目前越來越多的開源分布式處理系統如Cloudera、Apache Storm、Spark都支持與Kafka集成。

Kafka拓撲結構

安裝和啟動

Download

Download the 0.10.2.1 release and un-tar it.

tar -xzf kafka_2.11-0.10.2.1.tgz
cd kafka_2.11-0.10.2.1

Start the server

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don‘t already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance.

bin/zookeeper-server-start.sh config/zookeeper.properties

Now start the Kafka server:

bin/kafka-server-start.sh config/server.properties

Create a topic

bin/kafka-topics.sh --create --zookeeper localhost:8181 --replication-factor 1 --partitions 1 --topic test

查看topic

bin/kafka-topics.sh --list --zookeeper localhost:8181
test
 

Send some messages

bin/kafka-console-producer.sh --broker-list localhost:8092 --topic test

Start a consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:8092 --topic test --from-beginning

common commands

1. list topic command

   bin/kafka-topics.sh --list --zookeeper localhost:8181

2. 查詢topic

   bin/kafka-topics.sh --describe --zookeeper localhost:8181 --topic my-replicated-topic
   Topic:my-replicated-topic   PartitionCount:1    ReplicationFactor:3 Configs:
   Topic: my-replicated-topic  Partition: 0    Leader: 0   Replicas: 0,2,1 Isr: 0
    
   

參考

1. kafka學習筆記：知識點整理
2. Kafka入門經典教程

Kafka 入門