1. 程式人生 > >Kafka動態調整topic副本因子replication-factor

Kafka動態調整topic副本因子replication-factor

實際專案中我們可能在建立topic時沒有設定好正確的replication-factor,導致kafka叢集雖然是高可用的,但是該topic在有broker宕機時,可能發生無法使用的情況。topic一旦使用又不能輕易刪除重建,因此動態增加副本因子就成為最終的選擇。

原因分析:

假設我們有3個kafka broker分別brokerA、brokerB、brokerC.

  1. 當我們建立的topic有3個分割槽partition時並且replication-factor為1,基本上一個broker上一個分割槽。擋一個broker宕機了,該topic就無法使用了,因為三個分割槽只有兩個能用,
  2. 當我們建立的topic有3個分割槽partition時並且replication-factor為2時,可能分割槽資料分佈情況是
    brokerA, partiton0,partiton1,
    brokerB, partiton1,partiton2
    brokerC, partiton2,partiton0,
    每個分割槽有一個副本,當其中一個broker宕機了,kafka叢集還能完整湊出該topic的三個分割槽,例如當brokerA宕機了,可以通過brokerB和brokerC組合出topic的三個分割槽。

如何動態給已經建立的topic新增replication-factor?
可能很多人想使用kafka-topics.sh指令碼,那麼事情情況如何了?

[[email protected] bin]# ./kafka-topics.sh --alter --topic  yqtopic01  --zookeeper localhost:2181    --replication-factor 3
Option "[replication-factor]" can't be used with option"[alter]"
Option                                   Description
------                                   -----------
--alter                                  Alter the number of partitions,
                                           replica assignment, and/or
                                           configuration for
the topic. --config <String: name=value> A topic configuration override for the

截圖
在這裡插入圖片描述

可以看出kafka-topics.sh不能用來增加副本因子replication-factor。實際應該使用kafka bin目錄下面的kafka-reassign-partitions.sh
a, 首先我們配置topic的副本,儲存為json檔案()
例如, 我們想把yqtopic01的部分設定為3,(我的kafka叢集有3個broker,id分別為0,1,2), json檔名稱為increase-replication-factor.json

{“version”:1,
“partitions”:[
{“topic”:“yqtopic01”,“partition”:0,“replicas”:[0,1,2]},
{“topic”:“yqtopic01”,“partition”:1,“replicas”:[0,1,2]},
{“topic”:“yqtopic01”,“partition”:2,“replicas”:[0,1,2]}
]}
b, 然後執行指令碼
./kafka-reassign-partitions.sh -zookeeper 127.0.0.1:2181 --reassignment-json-file increase-replication-factor.json --execute

kafka-reassign-partitions.sh執行截圖
在這裡插入圖片描述

我們可以通過執行
kafka-topics.sh --describe --zookeeper localhost:2181 --topic yqtopic01檢視現在該topic的副本因子。

總結
所有文件官方文件最權威。https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor

摘錄如下:
Increasing replication factor
Increasing the replication factor of an existing partition is easy. Just specify the extra replicas in the custom reassignment json file and use it with the --execute option to increase the replication factor of the specified partitions.

For instance, the following example increases the replication factor of partition 0 of topic foo from 1 to 3. Before increasing the replication factor, the partition’s only replica existed on broker 5. As part of increasing the replication factor, we will add more replicas on brokers 6 and 7.

The first step is to hand craft the custom reassignment plan in a json file:

> cat increase-replication-factor.json
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

Then, use the json file with the --execute option to start the reassignment process:

	
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment
 
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}

 
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option:

	
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [foo,0] completed successfully

You can also verify the increase in replication factor with the kafka-topics tool:

	
> bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe
Topic:foo   PartitionCount:1    ReplicationFactor:3 Configs:
  Topic: foo    Partition: 0    Leader: 5   Replicas: 5,6,7 Isr: 5,6,7