kafka-重復消費-1
錯誤如下:
Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
造成的問題:假如consumer.properties配置中max.poll.records=40 (一次最多拉取40條數據) session.timeout.ms=30000 (會話時間)
假設kafka此時一次拉取了40條數據,但在處理第31條的時候拋出了如上的異常,就會導致,本次offset不會提交,完了這40條消息都會在接下來的某刻被再次消費,這其中就包含了其實已經消費了的30條數據
原因:the poll loop is spending too much time message processing, the time between subsequent calls to poll() was longer than the configured session.timeout.ms,好吧其實是一個意思!
意思就是說poll下來數據後,處理這些數據的時間比 session.timeout.ms配置的時間要長,從而導致the group has already rebalanced
解決辦法是最後一句話:You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
即要不增大 session.timeout.ms,要不減小max.poll.records ,至於具體配置為多少,得看你處理一條消息花費多長時間 x,需要滿足 x乘以max.poll.records < session.timeout.ms
kafka-重復消費-1