storm trident 一個batch多大
阿新 • • 發佈:2018-05-27
conf increase trident eas example part broker 有一個 storm
You can increase the batch size by changing "tridentKafkaConfig.fetchSizeBytes" property.
Also, batch size is related with number of brokers and number of partitions.
For example, if you have 2 brokers and 3 partitions for each broker that means the total count of partition is 6.
y this way, the batch size equals to tridentKafkaConfig.fetchSizeBytes X total partition count.
if we assume that the tridentKafkaConfig.fetchSizeBytes is 1024X1024, the batch size equals to 6 MB.(3x2x1024x1024)bytes
相當於從kafka裏面是批量讀取的,然後形成storm中的一個batch
而且spout裏面有一個buffer,應該也算是一個緩存,將正在處理的一部分數據,存在內存中,直到處理完
storm trident 一個batch多大