kafka文件(13)----0.10.1-Document-文件(5)-configures-consumer配置資訊
In 0.9.0.0 we introduced the new Java consumer as a replacement for the older Scala-based simple and high-level consumers. The configs for both new and old consumers are described below.
3.3 Consumer 配置
0.9.0.0版本中,引入了新的java版本的consumer,用來替代老的Scala版本以及高水位的consumers。因此,配置分為新舊兩種,具體如下所示:
Below is the configuration for the new consumer:
3.3.1 新Consumer配置
下面是新consumer的配置
bootstrap.servers |
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover
the full set of servers. This list should be in the form host/port對的列表,用來建立與kafka的初始連結。客戶端將使用列表中所有指定的servers-這個列表隻影響客戶端的初始化,客戶端需要使用這個列表去查詢所有servers的完整列表。列表格式應該為:host1:port1,host2,port2,....;因為這些server列表只是用來初始化發現完整的server列表(而完整的server列表可能在使用中發生變化,機器損壞,部署遷移等),這個表不需要包含所有server的ip和port(但是最好多於1個,預防這個server掛掉的風險,防止下次啟動無法連結) |
list | high | ||
key.deserializer |
Deserializer class for key that implements the Deserializer介面的金鑰的類的key |
class | high | ||
value.deserializer |
Deserializer class for value that implements the 實現Deserializer介面的Deserializer的值。 |
class | high | ||
fetch.min.bytes |
The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. server返回給抓取請求的最小資料量。在返回訊息量小於這個值時,請求會一直等待。預設設定為1,即只要有一個位元組就可以立刻返回,而不用等到超時。增大這個值,在一定程度上可以改善吞吐量,但是有可能帶來額外的延遲。 |
int | 1 | [0,...] | high |
group.id |
A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using 代表consumer組的唯一字串。當consumer通過subscribe(topic)或者基於kafka的offset管理策略來使用group管理函式時,必須要有group.id |
string | "" | high | |
heartbeat.interval.ms |
The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group.
The value must be set lower than 當使用Kafka的group管理用法時,consumer協作器兩次心跳之間的時間間隔。心跳連結用來保證consumer的會話依然活躍,以及在新consumer加入consumer group時可以重新進行負載均衡。這個值要比session.timeout.ms小,但是一般要比session.timeout.ms的1/3要打。這個值可以適當的減小,以控制重負載均衡的時間。 |
int | 3000 | high | |
max.partition.fetch.bytes |
The maximum amount of data per-partition the server will return. If the first message in the first non-empty partition of the fetch is larger than this limit, the message will still be returned to ensure that the consumer can make progress. The maximum message
size accepted by the broker is defined via server返回訊息中針對每個partition資料請求的最大資料量。這個值也不是絕對的,如果請求的第一個非空partition的第一條訊息大於這個值,則訊息依然會返回給consumer,以保證繼續進行。broker可以接受的訊息尺寸通過message.max.bytes(broker配置)或者max.message.bytes(topic配置)來設定。檢視fetch.max.bytes獲取consumer請求的最大訊息尺寸。 |
int | 1048576 | [0,...] | high |
session.timeout.ms |
The timeout used to detect consumer failures when using Kafka's group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout,
then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by 當使用Kafka group管理用法時,這個超時時間用來檢測consumer是否失效。consumer通過傳送心跳資訊給broker,用來表明自己還有效。如果broker在這個超時時間內沒有收到來自consumer的心跳資訊,則broker會從consumer group中移除這個consumer,並重新進行負載均衡。注意,這個值必須在broker配置的允許範圍之內:即group.min.session.timeout.ms和group.max.session.timeout.ms之間。 |
int | 10000 | high | |
ssl.key.password |
The password of the private key in the key store file. This is optional for client. 儲存在金鑰檔案中私有金鑰。這個是可選的 |
password | null | high | |
ssl.keystore.location | The location of the key store file. This is optional for client and can be used for two-way authentication for client. | string | null | high | |
ssl.keystore.password |
The store password for the key store file. This is optional for client and only needed if ssl.keystore.location is configured. 金鑰檔案路徑。這個是可選的 |
password | null | high | |
ssl.truststore.location |
The location of the trust store file. 受信任檔案的位置 |
string | null | high | |
ssl.truststore.password |
The password for the trust store file. 受信任檔案的密碼 |
password | null | high | |
auto.offset.reset |
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted): 當kafka沒有初始offset或者server中也不存在任何初始化offset時,consumer遇到這種情況應該從哪裡開始獲取訊息
|
string | latest | [latest, earliest, none] | medium |
connections.max.idle.ms |
Close idle connections after the number of milliseconds specified by this config. 空閒連結的超時時間:server socket處理執行緒會關閉超時的連結。 |
long | 540000 | medium | |
enable.auto.commit |
If true the consumer's offset will be periodically committed in the background. 如果設定為true,則consumer的offset會在後臺週期性的上傳 |
boolean | true | medium | |
exclude.internal.topics |
Whether records from internal topics (such as offsets) should be exposed to the consumer. If set to 內部topics(例如offsets)是否需要暴漏給consumer。如果設定true,從內部topic獲取資料的方式只能是訂閱它。 |
boolean | true | medium | |
fetch.max.bytes |
The maximum amount of data the server should return for a fetch request. This is not an absolute maximum, if the first message in the first non-empty partition of the fetch is larger than this value, the message will still be returned to ensure that the
consumer can make progress. The maximum message size accepted by the broker is defined via server針對抓取請求的應答中所包含的最大位元組數。但是這個值並不是絕對的,如果請求中第一個非空partition的第一條訊息大於這個值,這條訊息仍然會返回給客戶端,以保證繼續進行。broker可以接受的最大訊息尺寸通過message.max.bytes(broker config)或者max.message.bytes(topic config)確定。注意,consumer可以同時釋出多條請求資訊。 |
int | 52428800 | [0,...] | medium |
max.poll.interval.ms |
The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. 當使用consumer 組管理時,在兩次呼叫poll()之間的停留時間。這個值指明瞭consumer在抓取更多訊息之前的處於空閒的最長時間。如果poll()在這個值指定的超時之前沒有呼叫,則consumer會被認定為失效,consumer group會重新負載均衡,並重新分配失效consumer負責的partitions給其它consumer成員。 |
int | 300000 | [1,...] | medium |
max.poll.records |
The maximum number of records returned in a single call to poll(). 一次單獨呼叫poll()可以返回的訊息的最大條數。 |
int | 500 | [1,...] | medium |
partition.assignment.strategy |
The class name of the partition assignment strategy that the client will use to distribute partition ownership amongst consumer instances when group management is used partitions的分配策略的類名。當使用group管理策略時,客戶端用來將來將partitions分配給組中consumer例項。 |
list | [class org.apache.kafka.clients.consumer.RangeAssignor] | medium | |
receive.buffer.bytes |
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. TCP接受快取的大小(SO_RCVBUF)。如果設定為-1,則使用OS預設值. |
int | 65536 | [-1,...] | medium |
request.timeout.ms |
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. 客戶端等待broker應答的超時時間。如果超時了,客戶端沒有收到應答,如果必要的話可能會重發請求,如果重試都失敗了也可能會報請求失敗 |
int | 305000 | [0,...] | medium |
sasl.kerberos.service.name |
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. kafka執行的Kerberos主機名。可以在Kafka's JAAS配置或者Kafka's 配置中定義。 |
string | null | medium | |
sasl.mechanism |
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism. 客戶端連結進行通訊的SASL機制。預設時GSSAPI |
string | GSSAPI | medium | |
security.protocol |
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. brokers之間通訊使用的安全協議。正確值為:PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. |
string | PLAINTEXT | medium | |
send.buffer.bytes |
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. TCP傳送的socket的SO_SNDBUF快取。如果設定為-1,將使用OS的預設值 |
int | 131072 | [-1,...] | medium |
ssl.enabled.protocols |
The list of protocols enabled for SSL connections. SSL連結的協議 |
list | [TLSv1.2, TLSv1.1, TLSv1] | medium | |
ssl.keystore.type |
The file format of the key store file. This is optional for client. 金鑰檔案的檔案格式。對客戶端來說是可選的。 |
string | JKS | medium | |
ssl.protocol |
The SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. 生成SSLContext的SSL協議。預設配置時TLS,適用於大部分情況。最近JVMS支援的協議包括:TLS,TLSv1.1,TLSv1.2. |
string | TLS | medium | |
ssl.provider |
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. SSL連結安全提供者名字。預設是JVM |
string | null | medium | |
ssl.truststore.type |
The file format of the trust store file. 受信任的檔案的檔案格式 |
string | JKS | medium | |
auto.commit.interval.ms |
The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if consumer offsets可以自動提交到kafka的頻率(微秒), 如果設定 |
int | 5000 | [0,...] | low |
check.crcs |
Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance. 自動檢查消費的訊息的CRC32.這個檢查保證了訊息在傳送過程中沒有損壞,或者在磁碟上沒有損壞。這個檢查有可能增加負擔,因此對效能要求比較高的情況可能禁用這個檢查。 |
boolean |
true | low | |
client.id |
An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. 請求中會附帶上id 字串,用來標識客戶端。目的是追蹤請求的來源,用於檢查某些請求是否來自非法ip/port。 |
string | "" | low | |
fetch.max.wait.ms |
The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes. server在應答抓取請求之前可以阻塞的最長時間,如果沒有足夠的訊息滿足fetch.min.bytes,server一般會阻塞這麼長的時間,以獲取足夠的訊息 |
int | 500 | [0,...] | low |
interceptor.classes |
A list of classes to use as interceptors. Implementing the 用作攔截器的類的列表。介面ConsumerInterceptor可以攔截部分訊息,以防它們傳送到kafka叢集。預設情況下沒有攔截器 |
list | null | low | |
metadata.max.age.ms |
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. 更新metadata的時間間隔,無論partition的leader是否發生變換或者topic其它的元資料是否發生變化。 |
long | 300000 | [0,...] | low |
metric.reporters |
A list of classes to use as metrics reporters. Implementing the 用於實現指標統計的類的列表。MetricReporter介面允許呼叫實現指標統計的外掛類。JmxReporter總是包含註冊JMX統計。 |
list | [] | low | |
metrics.num.samples |
The number of samples maintained to compute metrics. 維護計算指標的樣本數 |
int | 2 | [1,...] | low |
metrics.sample.window.ms |
The window of time a metrics sample is computed over. 度量樣本的計算的時長 |
long | 30000 | [0,...] | low |
reconnect.backoff.ms |
The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker. 重連給定host之前的等待時間。避免頻繁的重連某個host。這個backoff時間也設定了consumer請求broker的重試等待時間。 |
long | 50 | [0,...] | low |
retry.backoff.ms |
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. 重新發送失敗請求給某個topic partition之前的最長等待時間,避免極短時間內頻繁的重試。 |
long | 100 | [0,...] | low |
sasl.kerberos.kinit.cmd |
Kerberos kinit command path. Kerberos kinit命令路徑 |
string | /usr/bin/kinit | low | |
sasl.kerberos.min.time.before.relogin |
Login thread sleep time between refresh attempts. 在重試之間登陸執行緒的睡眠時間 |
long | 60000 | low | |
sasl.kerberos.ticket.renew.jitter |
Percentage of random jitter added to the renewal time. 新增到更新時間的隨機抖動的百分比。 |
double | 0.05 | low | |
sasl.kerberos.ticket.renew.window.factor |
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. 重新進行登入驗證重新整理之前,登入執行緒的睡眠時間 |
double | 0.8 | low | |
ssl.cipher.suites |
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported. 密碼套件列表。 這是一種集認證,加密,MAC和金鑰交換演算法一塊的命名組合,用於使用TLS或SSL網路協議協商網路連線的安全設定。 預設情況下,支援所有可用的密碼套件。 |
list | null | low | |
ssl.endpoint.identification.algorithm |
The endpoint identification algorithm to validate server hostname using server certificate. 端點標識演算法,使用伺服器證書驗證伺服器主機名。 |
string | null | low | |
ssl.keymanager.algorithm |
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. 金鑰管理器工廠用於SSL連線的演算法。 預設值是為Java虛擬機器配置的金鑰管理器工廠演算法 |
string | SunX509 | low | |
ssl.secure.random.implementation |
The SecureRandom PRNG implementation to use for SSL cryptography operations. 用於SSL加密操作的SecureRandom PRNG實現。 |
string | null | low | |
ssl.trustmanager.algorithm |
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. ssl連結信任管理者工廠的演算法。預設時JVM支援的演算法。 |
string | PKIX | low |
The essential old consumer configurations are the following:
基本的consumer配置是:
group.id
zookeeper.connect
group.id |
A string that uniquely identifies the group of consumer processes to which this consumer belongs. By setting the same group id multiple processes indicate that they are all part of the same consumer group. 代表consumer組的唯一字串。 當consumer通過subscribe(topic) 或者基於kafka的offset管理策略來 使用group管理函式時,必須要有group.id |
|
zookeeper.connect |
Specifies the ZooKeeper connection string in the form and port of a ZooKeeper server. To allow connecting through other ZooKeeper nodes when that ZooKeeper machine is down you can also specify multiple hosts in the form The server may also have a ZooKeeper chroot path as part of its ZooKeeper connection string which puts its data under some path in the global ZooKeeper namespace. If so the consumer should use the same chroot path in its connection string. For example to give a chroot path of string as
指定zookeeper的連線的字串,格式是hostname:port, 此處host和port都是zookeeper server的host和port, 為避免某個zookeeper 機器宕機之後失聯, 你可以指定多個hostname:port,使用逗號作為分隔: 此路徑用於存放他自己的資料,方式: |
|
consumer.id | null |
Generated automatically if not set. 如果不設定的話會自動產生 |
socket.timeout.ms | 30 * 1000 |
The socket timeout for network requests. The actual timeout set will be max.fetch.wait + socket.timeout.ms. 網路請求的超時限制。真實的超時限制是 max.fetch.wait+socket.timeout.ms |
socket.receive.buffer.bytes | 64 * 1024 |
The socket receive buffer for network requests 網路請求的socket的receive快取 |
fetch.message.max.bytes | 1024 * 1024 |
The number of bytes of messages to attempt to fetch for each topic-partition in each fetch request. These bytes will be read into memory for each partition, so this helps control the memory used by the consumer. The fetch request size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. 每次fetch請求中,針對每次fetch訊息的最大位元組數。 這些位元組將會讀到用於每個partition的記憶體中,因此, 此設定將會控制consumer所使用的memory大小。 這個fetch請求尺寸必須至少和server允許的 最大訊息尺寸相等,否則,producer可能傳送的 訊息尺寸大於consumer所能消耗的尺寸。 |
num.consumer.fetchers | 1 |
The number fetcher threads used to fetch data. 用於抓取資料的抓取執行緒數 |
auto.commit.enable | true |
If true, periodically commit to ZooKeeper the offset of messages already fetched by the consumer. This committed offset will be used when the process fails as the position from which the new consumer will begin. 如果為真,consumer所fetch的訊息的offset將會 自動的同步到zookeeper。這項提交的offset將在 程序掛掉時,由新的consumer使用 |
auto.commit.interval.ms | 60 * 1000 |
The frequency in ms that the consumer offsets are committed to zookeeper. consumer向zookeeper提交offset的頻率, 單位是微秒 |
queued.max.message.chunks | 2 |
Max number of message chunks buffered for consumption. Each chunk can be up to fetch.message.max.bytes. 用於快取訊息的最大數目,以供consumption。 每個chunk必須和fetch.message.max.bytes相同 |
rebalance.max.retries | 4 |
When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalance will fail and retry. This setting controls the maximum number of attempts before giving up. 當新的consumer加入到consumer group時, consumers集合試圖重新平衡分配到每個 consumer的partitions數目。如果consumers 集合改變了,當分配正在執行時, 這個重新平衡會失敗並重入。 這個配置控制了放棄前的最大重試次數 |
fetch.min.bytes | 1 |
The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. 每次fetch請求時,server應該返回的最小位元組數。 如果沒有足夠的資料返回,請求會等待, 直到足夠的資料才會返回。 |
fetch.wait.max.ms | 100 |
The maximum amount of time the server will block before answering the fetch reques t if there isn't sufficient data to immediately satisfy fetch.min.bytes 如果沒有足夠的資料能夠滿足fetch.min.bytes, 則此項配置是指在應答fetch請求之前, server會阻塞的最大時間。 |
rebalance.backoff.ms | 2000 |
Backoff time between retries during rebalance. If not set explicitly, the value in zookeeper.sync.time.ms is used. 在重試reblance之前backoff時間。 如果沒有設定,則使用zookeeper.sync.time.ms |
refresh.leader.backoff.ms | 200 |
Backoff time to wait before trying to determine the leader of a partition that has just lost its leader. 在試圖確定某個partition的leader是否失去 他的leader地位之前,需要等待的backoff時間 |
auto.offset.reset | largest |
What to do when there is no initial offset in ZooKeeper or if an offset is out of range: zookeeper中沒有初始化的offset時,如果offset是以下值的迴應: |
consumer.timeout.ms | -1 |
Throw a timeout exception to the consumer if no message is available for consumption after the specified interval 如果沒有訊息可用,即使等待特定的時間之後 也沒有,則丟擲超時異常 |
exclude.internal.topics | true |
Whether messages from internal topics (such as offsets) should be exposed to the consumer. 是否將內部topics的訊息暴露給consumer |
client.id | group id value |
The client id is a user-specified string sent in each request to help trace calls. It should logically identify the application making the request. 是使用者特定的字串,用來在每次請求中幫助跟蹤呼叫。 它應該可以在邏輯上確認產生這個請求的應用 |
zookeeper.session.timeout.ms | 6000 |
ZooKeeper session timeout. If the consumer fails to heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur. zookeeper 會話的超時限制。如果consumer在這段時間 內沒有向zookeeper傳送心跳資訊,則它會被認為掛掉了, 並且reblance將會產生 |
zookeeper.connection.timeout.ms | 6000 |
The max time that the client waits while establishing a connection to zookeeper. 客戶端在建立通zookeeper連線中的最大等待時間 |
zookeeper.sync.time.ms | 2000 |
How far a ZK follower can be behind a ZK leader ZK follower可以落後ZK leader的最大時間 |
offsets.storage | zookeeper |
Select where offsets should be stored (zookeeper or kafka). 用於存放offsets的地點: zookeeper或者kafka |
offsets.channel.backoff.ms | 1000 |
The backoff period when reconnecting the offsets channel or retrying failed offset fetch/commit requests. 重新連線offsets channel或者是重試失敗的offset的 fetch/commit請求的backoff時間 |
offsets.channel.socket.timeout.ms | 10000 |
Socket timeout when reading responses for offset fetch/commit requests. This timeout is also used for ConsumerMetadata requests that are used to query for the offset manager. 當讀取offset的fetch/commit請求迴應的socket 超時限制。此超時限制是被consumerMetadata 請求用來請求offset管理 |
offsets.commit.max.retries | 5 |
Retry the offset commit up to this many times on failure. This retry count only applies to offset commits during shut-down. It does not apply to commits originating from the auto-commit thread. It also does not apply to attempts to query for the offset coordinator before committing offsets. i.e., if a consumer metadata request fails for any reason, it will be retried and that retry does not count toward this limit. 失敗時重試offset commit的最大次數。這個重試 只應用於offset commits失敗的時候。它不適用從 自動提交執行緒提交,也不適用於在提交offsets之前 試圖請求offset協作器。例如,如果consumer 元資料 請求由於某些原因失敗,將會重試, 而且不會計入這個重試次數。 |
dual.commit.enabled | true |
If you are using "kafka" as offsets.storage, you can dual commit offsets to ZooKeeper (in addition to Kafka). This is required during migration from zookeeper-based offset storage to kafka-based offset storage. With respect to any given consumer group, it is safe to turn this off after all instances within that group have been migrated to the new version that commits offsets to the broker (instead of directly to ZooKeeper). 如果使用“kafka”作為offsets.storage,你可以二次提交offset到 zookeeper(還有一次是提交到kafka)。在zookeeper-based的 offset storage到kafka-based的offset storage遷移時, 這是必須的。對任意給定的consumer group來說, 比較安全的建議是當完成遷移之後就關閉這個選項 |
partition.assignment.strategy | range |
Select between the "range" or "roundrobin" strategy for assigning partitions to consumer streams. The round-robin partition assignor lays out all the available partitions and all the available consumer threads. It then proceeds to do a round-robin assignment from partition to consumer thread. If the subscriptions of all consumer instances are identical, then the partitions will be uniformly distributed. (i.e., the partition ownership counts will be within a delta of exactly one across all consumer threads.) Round-robin assignment is permitted only if: (a) Every topic has the same number of streams within a consumer instance (b) The set of subscribed topics is identical for every consumer instance within the group. Range partitioning works on a per-topic basis. For each topic, we lay out the available partitions in numeric order and the consumer threads in lexicographic order. We then divide the number of partitions by the total number of consumer streams (threads) to determine the number of partitions to assign to each consumer. If it does not evenly divide, then the first few consumers will have one extra partition. 在“range”和“roundrobin”策略之間選擇一種作為 分配partitions給consumer 資料流的策略; 迴圈的partition分配器分配所有可用的partitions以及 所有可用consumer 執行緒。它會將partition迴圈的 分配到consumer執行緒上。如果所有consumer例項的 訂閱都是確定的,則partitions的劃分是確定的分佈。 迴圈分配策略只有在以下條件滿足時才可以: (1)每個topic在每個consumer實力上都有同樣數量 的資料流。 (2)訂閱的topic的集合對於consumer group中每個 consumer例項來說都是確定的。 |
More details about consumer configuration can be found in the scala class kafka.consumer.ConsumerConfig
.