kafka-storm spout拉取資料問題offset out of range
131236985 [Thread-15-spout] WARN storm.kafka.KafkaUtils - Got fetch request with offset out of range: [14026]; retrying with default start offset time from configuration. configured start offset time: [-2]
131236985 [Thread-17-spout] WARN storm.kafka.PartitionManager - Using new offset: 511883
131236985 [Thread-15-spout] WARN storm.kafka.PartitionManager - Using new offset: 511888
131236986 [Thread-15-spout] WARN storm.kafka.KafkaUtils - Got fetch request with offset out of range: [14026]; retrying with default start offset time from configuration. configured start offset time: [-2]
解決辦法:
1.修改SpoutConfig.zkRoot 值
String zkRoot = “”;
SpoutConfig spoutConf = new SpoutConfig(hosts, topic, zkRoot, id);
2.將spoutConf.forceFromStart 設為 true;
原因:
storm啟動的時候,預設是從頭讀取-2,當kafkaspout的task出錯時候,會重新啟動,這時候offset就會從zk中最老的offset,但kafka資料只會儲存一段時間,從zk中讀取道德offset的資料可能已經不存在,超過現在的範圍。
kafkaspout讀取的3種選項
-2: 從最老的開始讀
-1: 從最近的開始讀
0: 從Zk中讀