遠端連結Kafka總是那麼個錯---Failed to send messages after 3 tries
如果是在同一臺機器(localhost),Kafka和Java Client工作正常,Producer和Consumer都能正常傳送和接收訊息,但是一旦部署到兩臺機器,則預設配置的話不能正常工作。會出現“kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries”的錯誤。
[2015-07-02 15:15:39,295] WARN Error while fetching metadata [{TopicMetadata for topic datadog-dev ->
No partition metadata for topic datadog-dev due to kafka.common.LeaderNotAvailableException}] for topic [datadog-dev]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2015-07-02 15:15:39,295] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: datadog-dev (kafka.producer.async.DefaultEventHandler)
[2015-07-02 15:15:39,399] WARN Error while fetching metadata [{TopicMetadata for topic datadog-dev ->
No partition metadata for topic datadog-dev due to kafka.common.LeaderNotAvailableException}] for topic [datadog-dev]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2015-07-02 15:15:39,399] ERROR Failed to send requests for topics datadog-dev with correlation ids in [9,16] (kafka.producer.async.DefaultEventHandler)
[2015-07-02 15:15:39,399] ERROR Error in handling batch of 4 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:547)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
解決方法其實很簡單,只需要在Kafka的配置檔案server.properties中,設定好主機名即可:
# Hostname the broker will bind to. If not set, the server will bind to all interfaceshost.name=queue-server1
究其原因,其實從註釋裡,我們可以知道,這是一個指定broker的地址(嚴格來說是所監聽的網路介面,或者網絡卡),同時,也可以看出它還和下面的屬性相關。
# Hostname the broker will advertise to producers and consumers. If not set, it uses the
# value for “host.name” if configured. Otherwise, it will use the value returned from
# java.net.InetAddress.getCanonicalHostName().
#advertised.host.name=
也就是說,producer和consumer是通過這個主機名(advertised.host.name)來連線broker的,而如果這個值沒有設定,則會使用上面的host.name的值,如果上面的host.name也沒有設定,則會使用java.net.InetAddress.getCanonicalHostName()獲取的值。
從Zookeeper中可以看出,預設的時候該broker是localhost,從其他機器訪問當然不可能成功。