1. 程式人生 > >建立topic時沒有設定好正確的replication-factor

建立topic時沒有設定好正確的replication-factor

參考文章:https://blog.csdn.net/russle/article/details/83421904


一、問題描述

實際專案中我們可能在建立topic時沒有設定好正確的replication-factor,(預設情況為1份),導致kafka叢集雖然是高可用的,但是該topic所在的broker宕機時,可能發生無法使用的情況。topic一旦使用又不能輕易刪除重建,因此動態增加副本因子就成為最終的選擇。

二、問題解決

首先本地編輯一個json檔案reassign.json如下:


/*
*# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state" 
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3. 
offsets.topic.replication.factor=3 
*/
//實際預設為1,即offsets.topic.replication.factor=1 //動態增加之後如下: {"version":1, "partitions":[ {"topic":"__consumer_offsets","partition":0,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":1,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":2,"replicas":[1,2,3]}, {"topic":"__consumer_offsets"
,"partition":3,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":4,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":5,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":6,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":7,"replicas":[1,2,3]}, {"topic"
:"__consumer_offsets","partition":8,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":9,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":10,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":11,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":12,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":13,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":14,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":15,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":16,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":17,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":18,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":19,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":20,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":21,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":22,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":23,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":24,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":25,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":26,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":27,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":28,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":29,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":30,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":31,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":32,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":33,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":34,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":35,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":36,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":37,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":38,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":39,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":40,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":41,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":42,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":43,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":44,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":45,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":46,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":47,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":48,"replicas":[1,2,3]}, {"topic":"__consumer_offsets","partition":49,"replicas":[1,2,3]} ]}

如下儲存為reassign.json檔案,然後在shell的kafka的bin目錄下執行如下命令

kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassign.json --execute

這樣,就增加了topic的副本數量;防止topic中的offset所在的broker宕機時候不能提供消費;

如果需要指定自身topic中offset的儲存replicas位置(broker位置),則需要將"topic":"__consumer_offsets"改為相應的"topic":“topic_xxx”


我犯的錯誤是:

Partition reassignment data file is empty

被坑的不清,最後發現是json檔案格式中最後一行有一個逗號“,”。這還是在別人發現的問題。特此標註。
如下是原始的reassign.json檔案
在這裡插入圖片描述

去掉之後,重新執行

kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassign.json --execute

topic的replication設定成功!
在這裡插入圖片描述