1. 程式人生 > >kafka 自定義分割槽分配和遷移

kafka 自定義分割槽分配和遷移

自定義分割槽分配和遷移

分割槽重新分配工具還可用於選擇性地將分割槽的副本移動到特定的代理集。當以這種方式使用時,假設使用者知道重新分配計劃並且不需要工具生成候選重新​​分配,有效地跳過 - 生成步驟並直接移動到--execute步驟

例如,以下示例將主題foo1的分割槽0移動到代理5,6,將主題foo2的分割槽1移動到代理2,3:

第一步是在json檔案中手工製作自定義重新分配計劃:

1

2

> cat custom-reassignment.json

{"version":1,"partitions":[{"topic":"foo1"

,"partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]}

然後,使用帶有--execute選項的json檔案來啟動重新分配過程:

1

2

3

4

6

7

8

9

10

11

12

13

14

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute

Current partition replica assignment

 

{"version":1,

"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},

              {"topic":"foo2","partition":1,"replicas":[3,4]}]

}

 

Save this to use as the --reassignment-json-

file option during rollback

Successfully started reassignment of partitions

{"version":1,

"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},

              {"topic":"foo2","partition":1,"replicas":[2,3]}]

}

--verify選項可與該工具一起使用,以檢查分割槽重新分配的狀態。請注意,相同的expand-cluster-reassignment.json(與--execute選項一起使用)應與--verify選項一起使用:

1

2

3

4

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify

Status of partition reassignment:

Reassignment of partition [foo1,0] completed successfully

Reassignment of partition [foo2,1] completed successfully

 

增加複製因子

增加現有分割槽的複製因子很容易。只需在自定義重新分配json檔案中指定額外副本,並將其與--execute選項一起使用,以增加指定分割槽的複製因子。

例如,以下示例將主題foo的分割槽0的複製因子從1增加到3.在增加複製因子之前,分割槽的唯一副本存在於代理5上。作為增加複製因子的一部分,我們將新增更多副本經紀人6和7。

第一步是在json檔案中手工製作自定義重新分配計劃:

1

2

3

> cat increase-replication-factor.json

{"version":1,

"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

然後,使用帶有--execute選項的json檔案來啟動重新分配過程:

1

2

3

4

6

7

8

9

10

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute

Current partition replica assignment

 

{"version":1,

"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}

 

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions

{"version":1,

"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

--verify選項可與該工具一起使用,以檢查分割槽重新分配的狀態。請注意,相同的increase-replication-factor.json(與--execute選項一起使用)應與--verify選項一起使用:

1

2

3

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify

Status of partition reassignment:

Reassignment of partition [foo,0] completed successfully

您還可以使用kafka-topics工具驗證複製因子的增加:

1

2

3

> bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe

Topic:foo   PartitionCount:1    ReplicationFactor:3 Configs:

  Topic: foo    Partition: 0    Leader: 5   Replicas: 5,6,7 Isr: 5,6,7