auto.offset.reset介紹及spark無法讀取屬性對於old message
阿新 • • 發佈:2018-11-30
我們先看看最新的官方文件說明:
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):
- earliest: automatically reset the offset to the earliest offset
- latest: automatically reset the offset to the latest offset
- none: throw exception to the consumer if no previous offset is found for the consumer's group
- anything else: throw exception to the consumer.
翻譯:
當Kafka中沒有初始偏移或如果當前偏移在伺服器上不再存在時(例如,因為該資料已被刪除),該怎麼辦:
- 最早:自動將偏移重置為最早的偏移
- 最新:自動將偏移重置為最新偏移
- none:如果沒有為消費者組找到以前的偏移,則向消費者丟擲異常
- 任何其他:丟擲異常到消費者。
“auto.offset.reset”的值只能是:[latest, earliest, none]中的一個,預設是"latest"
Please understand, Spark can't read old messages with property:
"auto.offset.reset" -> "latest"