技術標籤：kafka

問題概述

kafka程序不定期掛掉。ERROR Failed to clean up log for __consumer_offsets-30 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)，報錯如下

[2020-12-07 16:12:36,803] ERROR Failed to clean up log for __consumer_offsets-7 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.NoSuchFileException: /tmp/kafka-logs/__consumer_offsets-7/00000000000000000000.log
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
        at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:913)
        at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:227)
        at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:495)
        at kafka.log.Log.$anonfun$deleteSegmentFiles$1(Log.scala:2230)
        at kafka.log.Log.$anonfun$deleteSegmentFiles$1$adapted(Log.scala:2230)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.log.Log.deleteSegmentFiles(Log.scala:2230)
        at kafka.log.Log.$anonfun$replaceSegments$6(Log.scala:2300)
        at kafka.log.Log.$anonfun$replaceSegments$6$adapted(Log.scala:2295)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.log.Log.replaceSegments(Log.scala:2295)
        at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:606)
        at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:531)
        at kafka.log.Cleaner.doClean(LogCleaner.scala:530)
        at kafka.log.Cleaner.clean(LogCleaner.scala:504)
        at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:373)
        at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:345)
        at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:325)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:314)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
        Suppressed: java.nio.file.NoSuchFileException: /tmp/kafka-logs/__consumer_offsets-7/00000000000000000000.log -> /tmp/kafka-logs/__consumer_offsets-7/00000000000000000000.log.deleted
                at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
                at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
                at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:910)
                ... 19 more
 

問題分析

錯誤顯示沒找到檔案導致報錯。linux會定時清理/tmp目錄下的檔案，我的kafka日誌檔案目錄正是放在了/tmp/kafka-logs目錄下，導致被定時給清理掉了，所以kafka在嘗試讀取或追加日誌時就會出錯。

grep log.dirs /opt/kafka_2.12-2.3.0/config/server.properties
/tmp/kafka-logs

問題解決

第一種：修改 日誌目錄，然後重啟kafka

log.dirs=/opt/kafka_2.12-2.3.0/kafka-logs/

第二種：新增kafka日誌目錄到清理白名單中

centos7：centos7下/tmp目錄的清理由服務systemd負責，其相關配置檔案在/usr/lib/tmpfiles.d目錄下，我們修改配置檔案tmp.conf，將kafka日誌目錄加進去，

#防止刪除kafka日誌檔案
X /tmp/kafka-logs

centos6：centos6下/tmp目錄的清理是通過tmpwatch來實現的，tmpwatch則依賴於cron的定時排程，排程檔案為/etc/cron.daily/tmpwatch

#防止刪除kafka日誌檔案
X /tmp/kafka-logs