Storm 反壓機制 back pressure設定失效
topology.max.spout.pending 的意義在於 ,快取spout 傳送出去的tuple,當下流的bolt還有topology.max.spout.pending 個 tuple 沒有消費完時,spout會停下來,等待下游bolt去消費,當tuple 的個數少於topology.max.spout.pending個數時,spout 會繼續從訊息源讀取訊息。(這個屬性只對可靠訊息處理有用)。
This new feature is aimed for automatic flow control through the topology DAG since different components may have unmatched
tuple processing speed. Currently, the tuples may get dropped if the downstream components can not process as quickly, thereby causing a waste of network bandwidth and processing capability. In addition, it is difficult to tune the max.spout.pending parameter
for best backpressure performance. Another big motivation is that using max.spout.pending for flow control forces users to enable acking, which does not make sense for the scenarios where acking is not needed and flow control is needed (e.g., the at-most-once
cases). Therefore, an automatic back pressure scheme is highly desirable.
In this design, spouts throttle not only when max.spout.pending is hit, but also when any bolt has gone over a high water mark
in their receive queue, and has not yet gone below a low water mark again. There is a lot of room for potential improvement here around control theory and having spouts only respond to downstream bolts backing up, but a simple bang-bang controller like this
is a great start.
Our ABP scheme implements a light-weight yet efficient back pressure scheme. It monitors certain queues in executors and worker and exploits the callback schemes on ZooKeeper and disruptor queue for a fast-responding (in a push manner) flow control.
- worker executor的接收佇列大於高水位,通知反壓執行緒
- worker反壓執行緒通知zookeeper,executor繁忙事件
- 所有worker監聽zookeeper executor繁忙的事件
- worker spouts降低傳送tuple速度
原文:
在Alibaba的github上,其說到"storm 社群的人想通過動態調整max_pending達到這種效果,其實這種做法根本無效。"。Alibaba的JStorm中有其自己的解決方案。
在我的專案中,遇到反壓機制的時候,曾經按照社群的做法,設定那個引數,但是我們的經驗是,設定的這個引數有時候不一定有效,我們的做法是手動的刪除Backpressure Thread的同步鎖。