1. 程式人生 > >kafka文件(13)----0.10.1-Document-文件(5)-configures-consumer配置資訊

kafka文件(13)----0.10.1-Document-文件(5)-configures-consumer配置資訊

In 0.9.0.0 we introduced the new Java consumer as a replacement for the older Scala-based simple and high-level consumers. The configs for both new and old consumers are described below.

3.3 Consumer 配置

0.9.0.0版本中,引入了新的java版本的consumer,用來替代老的Scala版本以及高水位的consumers。因此,配置分為新舊兩種,具體如下所示:

Below is the configuration for the new consumer:

3.3.1 新Consumer配置

下面是新consumer的配置

NAME DESCRIPTION TYPE DEFAULT VALID VALUES IMPORTANCE
bootstrap.servers

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the formhost1:port1,host2:port2,...

. Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

host/port對的列表,用來建立與kafka的初始連結。客戶端將使用列表中所有指定的servers-這個列表隻影響客戶端的初始化,客戶端需要使用這個列表去查詢所有servers的完整列表。列表格式應該為:host1:port1,host2,port2,....;因為這些server列表只是用來初始化發現完整的server列表(而完整的server列表可能在使用中發生變化,機器損壞,部署遷移等),這個表不需要包含所有server的ip和port(但是最好多於1個,預防這個server掛掉的風險,防止下次啟動無法連結)


list high
key.deserializer

Deserializer class for key that implements theDeserializer interface.

Deserializer介面的金鑰的類的key

class high
value.deserializer

Deserializer class for value that implements theDeserializer interface.

實現Deserializer介面的Deserializer的值。

class high
fetch.min.bytes

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

server返回給抓取請求的最小資料量。在返回訊息量小於這個值時,請求會一直等待。預設設定為1,即只要有一個位元組就可以立刻返回,而不用等到超時。增大這個值,在一定程度上可以改善吞吐量,但是有可能帶來額外的延遲。

int 1 [0,...] high
group.id

A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy.

代表consumer組的唯一字串。當consumer通過subscribe(topic)或者基於kafka的offset管理策略來使用group管理函式時,必須要有group.id

string "" high
heartbeat.interval.ms

The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower thansession.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

當使用Kafka的group管理用法時,consumer協作器兩次心跳之間的時間間隔。心跳連結用來保證consumer的會話依然活躍,以及在新consumer加入consumer group時可以重新進行負載均衡。這個值要比session.timeout.ms小,但是一般要比session.timeout.ms的1/3要打。這個值可以適當的減小,以控制重負載均衡的時間。

int 3000 high
max.partition.fetch.bytes

The maximum amount of data per-partition the server will return. If the first message in the first non-empty partition of the fetch is larger than this limit, the message will still be returned to ensure that the consumer can make progress. The maximum message size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size

server返回訊息中針對每個partition資料請求的最大資料量。這個值也不是絕對的,如果請求的第一個非空partition的第一條訊息大於這個值,則訊息依然會返回給consumer,以保證繼續進行。broker可以接受的訊息尺寸通過message.max.bytes(broker配置)或者max.message.bytes(topic配置)來設定。檢視fetch.max.bytes獲取consumer請求的最大訊息尺寸。

int 1048576 [0,...] high
session.timeout.ms

The timeout used to detect consumer failures when using Kafka's group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration bygroup.min.session.timeout.ms andgroup.max.session.timeout.ms.

當使用Kafka group管理用法時,這個超時時間用來檢測consumer是否失效。consumer通過傳送心跳資訊給broker,用來表明自己還有效。如果broker在這個超時時間內沒有收到來自consumer的心跳資訊,則broker會從consumer group中移除這個consumer,並重新進行負載均衡。注意,這個值必須在broker配置的允許範圍之內:即group.min.session.timeout.ms和group.max.session.timeout.ms之間。

int 10000 high
ssl.key.password

The password of the private key in the key store file. This is optional for client.

儲存在金鑰檔案中私有金鑰。這個是可選的

password null high
ssl.keystore.location The location of the key store file. This is optional for client and can be used for two-way authentication for client. string null high
ssl.keystore.password

The store password for the key store file. This is optional for client and only needed if ssl.keystore.location is configured.

金鑰檔案路徑。這個是可選的

password null high
ssl.truststore.location

The location of the trust store file.

受信任檔案的位置

string null high
ssl.truststore.password

The password for the trust store file.

受信任檔案的密碼

password null high
auto.offset.reset

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted): 當kafka沒有初始offset或者server中也不存在任何初始化offset時,consumer遇到這種情況應該從哪裡開始獲取訊息

  • earliest: automatically reset the offset to the earliest offset自動設定offset為最早的offset
  • latest: automatically reset the offset to the latest offset自動設定offset為最新的offset
  • none: throw exception to the consumer if no previous offset is found for the consumer's group如果沒有發現此consumer的offset值,則丟擲異常
  • anything else: throw exception to the consumer.丟擲異常
string latest [latest, earliest, none] medium
connections.max.idle.ms

Close idle connections after the number of milliseconds specified by this config.

空閒連結的超時時間:server socket處理執行緒會關閉超時的連結。

long 540000 medium
enable.auto.commit

If true the consumer's offset will be periodically committed in the background.

如果設定為true,則consumer的offset會在後臺週期性的上傳

boolean true medium
exclude.internal.topics

Whether records from internal topics (such as offsets) should be exposed to the consumer. If set to true the only way to receive records from an internal topic is subscribing to it.

內部topics(例如offsets)是否需要暴漏給consumer。如果設定true,從內部topic獲取資料的方式只能是訂閱它。

boolean true medium
fetch.max.bytes

The maximum amount of data the server should return for a fetch request. This is not an absolute maximum, if the first message in the first non-empty partition of the fetch is larger than this value, the message will still be returned to ensure that the consumer can make progress. The maximum message size accepted by the broker is defined via message.max.bytes (broker config) ormax.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel.

server針對抓取請求的應答中所包含的最大位元組數。但是這個值並不是絕對的,如果請求中第一個非空partition的第一條訊息大於這個值,這條訊息仍然會返回給客戶端,以保證繼續進行。broker可以接受的最大訊息尺寸通過message.max.bytes(broker config)或者max.message.bytes(topic config)確定。注意,consumer可以同時釋出多條請求資訊。

int 52428800 [0,...] medium
max.poll.interval.ms

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member.

當使用consumer 組管理時,在兩次呼叫poll()之間的停留時間。這個值指明瞭consumer在抓取更多訊息之前的處於空閒的最長時間。如果poll()在這個值指定的超時之前沒有呼叫,則consumer會被認定為失效,consumer group會重新負載均衡,並重新分配失效consumer負責的partitions給其它consumer成員。

int 300000 [1,...] medium
max.poll.records

The maximum number of records returned in a single call to poll().

一次單獨呼叫poll()可以返回的訊息的最大條數。

int 500 [1,...] medium
partition.assignment.strategy

The class name of the partition assignment strategy that the client will use to distribute partition ownership amongst consumer instances when group management is used

partitions的分配策略的類名。當使用group管理策略時,客戶端用來將來將partitions分配給組中consumer例項。

list [class org.apache.kafka.clients.consumer.RangeAssignor] medium
receive.buffer.bytes

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.

TCP接受快取的大小(SO_RCVBUF)。如果設定為-1,則使用OS預設值.

int 65536 [-1,...] medium
request.timeout.ms

The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

客戶端等待broker應答的超時時間。如果超時了,客戶端沒有收到應答,如果必要的話可能會重發請求,如果重試都失敗了也可能會報請求失敗

int 305000 [0,...] medium
sasl.kerberos.service.name

The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

kafka執行的Kerberos主機名。可以在Kafka's JAAS配置或者Kafka's 配置中定義。

string null medium
sasl.mechanism

SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

客戶端連結進行通訊的SASL機制。預設時GSSAPI

string GSSAPI medium
security.protocol

Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

brokers之間通訊使用的安全協議。正確值為:PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

string PLAINTEXT medium
send.buffer.bytes

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.

TCP傳送的socket的SO_SNDBUF快取。如果設定為-1,將使用OS的預設值

int 131072 [-1,...] medium
ssl.enabled.protocols

The list of protocols enabled for SSL connections.

SSL連結的協議

list [TLSv1.2, TLSv1.1, TLSv1] medium
ssl.keystore.type

The file format of the key store file. This is optional for client.

金鑰檔案的檔案格式。對客戶端來說是可選的。

string JKS medium
ssl.protocol

The SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities.

生成SSLContext的SSL協議。預設配置時TLS,適用於大部分情況。最近JVMS支援的協議包括:TLS,TLSv1.1,TLSv1.2.
SSL,SSLv2,SSLv3在老版本的JVMS中可用,但是由於知名的安全漏洞,它們並不受歡迎。

string TLS medium
ssl.provider

The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

SSL連結安全提供者名字。預設是JVM

string null medium
ssl.truststore.type

The file format of the trust store file.

受信任的檔案的檔案格式

string JKS medium
auto.commit.interval.ms

The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commitis set to true.

consumer offsets可以自動提交到kafka的頻率(微秒), 如果設定

int 5000 [0,...] low
check.crcs

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

自動檢查消費的訊息的CRC32.這個檢查保證了訊息在傳送過程中沒有損壞,或者在磁碟上沒有損壞。這個檢查有可能增加負擔,因此對效能要求比較高的情況可能禁用這個檢查。

boolean

true low
client.id

An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

請求中會附帶上id 字串,用來標識客戶端。目的是追蹤請求的來源,用於檢查某些請求是否來自非法ip/port。

string "" low
fetch.max.wait.ms

The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

server在應答抓取請求之前可以阻塞的最長時間,如果沒有足夠的訊息滿足fetch.min.bytes,server一般會阻塞這麼長的時間,以獲取足夠的訊息

int 500 [0,...] low
interceptor.classes

A list of classes to use as interceptors. Implementing theConsumerInterceptor interface allows you to intercept (and possibly mutate) records received by the consumer. By default, there are no interceptors.

用作攔截器的類的列表。介面ConsumerInterceptor可以攔截部分訊息,以防它們傳送到kafka叢集。預設情況下沒有攔截器

list null low
metadata.max.age.ms

The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

更新metadata的時間間隔,無論partition的leader是否發生變換或者topic其它的元資料是否發生變化。

long 300000 [0,...] low
metric.reporters

A list of classes to use as metrics reporters. Implementing the MetricReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

用於實現指標統計的類的列表。MetricReporter介面允許呼叫實現指標統計的外掛類。JmxReporter總是包含註冊JMX統計。

list [] low
metrics.num.samples

The number of samples maintained to compute metrics.

維護計算指標的樣本數

int 2 [1,...] low
metrics.sample.window.ms

The window of time a metrics sample is computed over.

度量樣本的計算的時長

long 30000 [0,...] low
reconnect.backoff.ms

The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker.

重連給定host之前的等待時間。避免頻繁的重連某個host。這個backoff時間也設定了consumer請求broker的重試等待時間。

long 50 [0,...] low
retry.backoff.ms

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

重新發送失敗請求給某個topic partition之前的最長等待時間,避免極短時間內頻繁的重試。

long 100 [0,...] low
sasl.kerberos.kinit.cmd

Kerberos kinit command path.

Kerberos kinit命令路徑

string /usr/bin/kinit low
sasl.kerberos.min.time.before.relogin

Login thread sleep time between refresh attempts.

在重試之間登陸執行緒的睡眠時間

long 60000 low
sasl.kerberos.ticket.renew.jitter

Percentage of random jitter added to the renewal time.

新增到更新時間的隨機抖動的百分比。

double 0.05 low
sasl.kerberos.ticket.renew.window.factor

Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

重新進行登入驗證重新整理之前,登入執行緒的睡眠時間

double 0.8 low
ssl.cipher.suites

A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

密碼套件列表。 這是一種集認證,加密,MAC和金鑰交換演算法一塊的命名組合,用於使用TLS或SSL網路協議協商網路連線的安全設定。 預設情況下,支援所有可用的密碼套件。

list null low
ssl.endpoint.identification.algorithm

The endpoint identification algorithm to validate server hostname using server certificate.

端點標識演算法,使用伺服器證書驗證伺服器主機名。

string null low
ssl.keymanager.algorithm

The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

金鑰管理器工廠用於SSL連線的演算法。 預設值是為Java虛擬機器配置的金鑰管理器工廠演算法

string SunX509 low
ssl.secure.random.implementation

The SecureRandom PRNG implementation to use for SSL cryptography operations.

用於SSL加密操作的SecureRandom PRNG實現。

string null low
ssl.trustmanager.algorithm

The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

ssl連結信任管理者工廠的演算法。預設時JVM支援的演算法。

string PKIX low

The essential old consumer configurations are the following:

基本的consumer配置是:

  • group.id
  • zookeeper.connect
PROPERTY DEFAULT DESCRIPTION
group.id

A string that uniquely identifies the group of consumer 

processes to which this consumer belongs. By setting 

the same group id multiple processes indicate that they

 are all part of the same consumer group.

代表consumer組的唯一字串。

當consumer通過subscribe(topic)

或者基於kafka的offset管理策略來

使用group管理函式時,必須要有group.id

zookeeper.connect

Specifies the ZooKeeper connection string in the form

 hostname:port where host and port are the host 

and port of a ZooKeeper server. To allow connecting 

through other ZooKeeper nodes when that ZooKeeper

 machine is down you can also specify multiple hosts

 in the form

 hostname1:port1,hostname2:port2,hostname3:port3.

The server may also have a ZooKeeper chroot path as 

part of its ZooKeeper connection string which puts its data

 under some path in the global ZooKeeper namespace.

 If so the consumer should use the same chroot path

 in its connection string. For example to give a chroot

 path of /chroot/path you would give the connection 

string as

hostname1:port1,hostname2:port2,hostname3:port3/chroot/path.

指定zookeeper的連線的字串,格式是hostname:port,

此處host和port都是zookeeper server的host和port,

為避免某個zookeeper 機器宕機之後失聯,

你可以指定多個hostname:port,使用逗號作為分隔:
hostname1:port1,hostname2:port2,hostname3:port3
可以在zookeeper連線字串中加入zookeeper的chroot路徑,

此路徑用於存放他自己的資料,方式:
hostname1:port1,hostname2:port2,hostname3:port3/chroot/path

consumer.id null

Generated automatically if not set.

如果不設定的話會自動產生

socket.timeout.ms 30 * 1000

The socket timeout for network requests. The actual timeout

 set will be max.fetch.wait + socket.timeout.ms.

網路請求的超時限制。真實的超時限制是 

  max.fetch.wait+socket.timeout.ms

socket.receive.buffer.bytes 64 * 1024

The socket receive buffer for network requests

網路請求的socket的receive快取

fetch.message.max.bytes 1024 * 1024

The number of bytes of messages to attempt to fetch for 

each topic-partition in each fetch request. These bytes will 

be read into memory for each partition, so this helps control

 the memory used by the consumer. The fetch request size

 must be at least as large as the maximum message size 

the server allows or else it is possible for the producer to 

send messages larger than the consumer can fetch.

每次fetch請求中,針對每次fetch訊息的最大位元組數。

這些位元組將會讀到用於每個partition的記憶體中,因此,

此設定將會控制consumer所使用的memory大小。

這個fetch請求尺寸必須至少和server允許的

最大訊息尺寸相等,否則,producer可能傳送的

訊息尺寸大於consumer所能消耗的尺寸。

num.consumer.fetchers 1

The number fetcher threads used to fetch data.

用於抓取資料的抓取執行緒數

auto.commit.enable true

If true, periodically commit to ZooKeeper the offset 

of messages already fetched by the consumer. 

This committed offset will be used when the process fails 

as the position from which the new consumer will begin.

如果為真,consumer所fetch的訊息的offset將會

自動的同步到zookeeper。這項提交的offset將在

程序掛掉時,由新的consumer使用

auto.commit.interval.ms 60 * 1000

The frequency in ms that the consumer offsets

 are committed to zookeeper.

consumer向zookeeper提交offset的頻率,

單位是微秒

queued.max.message.chunks 2

Max number of message chunks buffered for consumption. 

Each chunk can be up to fetch.message.max.bytes.

用於快取訊息的最大數目,以供consumption。

每個chunk必須和fetch.message.max.bytes相同

rebalance.max.retries 4

When a new consumer joins a consumer group the set 

of consumers attempt to "rebalance" the load to assign 

partitions to each consumer. If the set of consumers 

changes while this assignment is taking place the 

rebalance will fail and retry. This setting controls the 

maximum number of attempts before giving up.

當新的consumer加入到consumer  group時,

consumers集合試圖重新平衡分配到每個

consumer的partitions數目。如果consumers

集合改變了,當分配正在執行時,

這個重新平衡會失敗並重入。

這個配置控制了放棄前的最大重試次數

fetch.min.bytes 1

The minimum amount of data the server should 

return for a fetch request. If insufficient data is 

available the request will wait for that much data to 

accumulate before answering the request.

每次fetch請求時,server應該返回的最小位元組數。

如果沒有足夠的資料返回,請求會等待,

直到足夠的資料才會返回。

fetch.wait.max.ms 100

The maximum amount of time the server will 

block before answering the fetch reques

t if there isn't sufficient data to immediately 

satisfy fetch.min.bytes

如果沒有足夠的資料能夠滿足fetch.min.bytes,

則此項配置是指在應答fetch請求之前,

server會阻塞的最大時間。

rebalance.backoff.ms 2000

Backoff time between retries during rebalance. 

If not set explicitly, the value in

 zookeeper.sync.time.ms is used.

在重試reblance之前backoff時間。

如果沒有設定,則使用zookeeper.sync.time.ms

refresh.leader.backoff.ms 200

Backoff time to wait before trying to determine the 

leader of a partition that has just lost its leader.

在試圖確定某個partition的leader是否失去

他的leader地位之前,需要等待的backoff時間

auto.offset.reset largest

What to do when there is no initial offset in ZooKeeper 

or if an offset is out of range:
* smallest : automatically reset the offset to the smallest offset
* largest : automatically reset the offset to the largest offset
* anything else: throw exception to the consumer

zookeeper中沒有初始化的offset時,如果offset是以下值的迴應:
smallest:自動復位offset為smallest的offset
largest:自動復位offset為largest的offset
anything  else:向consumer丟擲異常

consumer.timeout.ms -1

Throw a timeout exception to the consumer if no message 

is available for consumption after the specified interval

如果沒有訊息可用,即使等待特定的時間之後

也沒有,則丟擲超時異常

exclude.internal.topics true

Whether messages from internal topics (such as offsets) 

should be exposed to the consumer.

是否將內部topics的訊息暴露給consumer

client.id group id value

The client id is a user-specified string sent in each request

 to help trace calls. It should logically identify the application

 making the request.

是使用者特定的字串,用來在每次請求中幫助跟蹤呼叫。

它應該可以在邏輯上確認產生這個請求的應用

zookeeper.session.timeout.ms  6000

ZooKeeper session timeout. If the consumer fails to heartbeat

 to ZooKeeper for this period of time it is considered dead 

and a rebalance will occur.

zookeeper 會話的超時限制。如果consumer在這段時間

內沒有向zookeeper傳送心跳資訊,則它會被認為掛掉了,

並且reblance將會產生

zookeeper.connection.timeout.ms 6000

The max time that the client waits while establishing a

 connection to zookeeper.

客戶端在建立通zookeeper連線中的最大等待時間

zookeeper.sync.time.ms  2000

How far a ZK follower can be behind a ZK leader

ZK follower可以落後ZK leader的最大時間

offsets.storage zookeeper

Select where offsets should be stored (zookeeper or kafka).

用於存放offsets的地點: zookeeper或者kafka

offsets.channel.backoff.ms 1000

The backoff period when reconnecting the offsets channel 

or retrying failed offset fetch/commit requests.

重新連線offsets channel或者是重試失敗的offset的

fetch/commit請求的backoff時間

offsets.channel.socket.timeout.ms 10000

Socket timeout when reading responses for offset 

fetch/commit requests. This timeout is also used for 

ConsumerMetadata requests that are used to query 

for the offset manager.

當讀取offset的fetch/commit請求迴應的socket 

超時限制。此超時限制是被consumerMetadata

請求用來請求offset管理

offsets.commit.max.retries 5

Retry the offset commit up to this many times on failure. 

This retry count only applies to offset commits during 

shut-down. It does not apply to commits originating 

from the auto-commit thread. It also does not apply to 

attempts to query for the offset coordinator before 

committing offsets. i.e., if a consumer metadata 

request fails for any reason, it will be retried and that 

retry does not count toward this limit.

失敗時重試offset commit的最大次數。這個重試

只應用於offset  commits失敗的時候。它不適用從

自動提交執行緒提交,也不適用於在提交offsets之前

試圖請求offset協作器。例如,如果consumer 元資料

請求由於某些原因失敗,將會重試,

而且不會計入這個重試次數。

dual.commit.enabled true

If you are using "kafka" as offsets.storage, you can 

dual commit offsets to ZooKeeper (in addition to Kafka). 

This is required during migration from zookeeper-based 

offset storage to kafka-based offset storage. With 

respect to any given consumer group, it is

 safe to turn this off after all instances within that 

group have been migrated to the new

 version that commits offsets to the broker

 (instead of directly to ZooKeeper).

如果使用“kafka”作為offsets.storage,你可以二次提交offset到

zookeeper(還有一次是提交到kafka)。在zookeeper-based的

offset  storage到kafka-based的offset storage遷移時,

這是必須的。對任意給定的consumer  group來說,

比較安全的建議是當完成遷移之後就關閉這個選項

partition.assignment.strategy range

Select between the "range" or "roundrobin" strategy 

for assigning partitions to consumer streams.

The round-robin partition assignor lays out all the

 available partitions and all the available consumer 

threads. It then proceeds to do a round-robin 

assignment from partition to consumer thread. 

If the subscriptions of all consumer instances are identical, 

then the partitions will be uniformly distributed. 

(i.e., the partition ownership counts will be within a 

delta of exactly one across all consumer threads.) 

Round-robin assignment is permitted only if:

 (a) Every topic has the same number of streams within

 a consumer instance (b) The set of subscribed topics 

is identical for every consumer instance within the group.

Range partitioning works on a per-topic basis. 

For each topic, we lay out the available partitions 

in numeric order and the consumer threads in 

lexicographic order. We then divide the number of 

partitions by the total number of consumer streams (threads) to 

determine the number of partitions to assign to each 

consumer. If it does not evenly divide, then the first 

few consumers will have one extra partition.

在“range”和“roundrobin”策略之間選擇一種作為

分配partitions給consumer 資料流的策略; 

迴圈的partition分配器分配所有可用的partitions以及

所有可用consumer  執行緒。它會將partition迴圈的

分配到consumer執行緒上。如果所有consumer例項的

訂閱都是確定的,則partitions的劃分是確定的分佈。

迴圈分配策略只有在以下條件滿足時才可以:

(1)每個topic在每個consumer實力上都有同樣數量

的資料流。

(2)訂閱的topic的集合對於consumer  group中每個

consumer例項來說都是確定的。

More details about consumer configuration can be found in the scala class kafka.consumer.ConsumerConfig.