執行flume windows版本時遇到如下錯誤：

2013-02-25 12:05:37,818 (pool-4-thread-1) [INFO - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(Spooling
FileLineReader.java:167)] Last read was never comitted - resetting mark position.
2013-02-25 12:05:41,297 (pool-5-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.r
un(SpoolDirectorySource.java:148)] Uncaught exception in Runnable
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name:
 memch1}
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
        at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:143)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.j
ava:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.
java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keepin
g up with sources, or the buffer size is too tight
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 10 more
2013-02-25 12:05:41,874 (pool-5-thread-1) [INFO - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(Spooling
FileLineReader.java:167)] Last read was never comitted - resetting mark position.
 

source為spooldir,sink 為avro。 我們在執行flume時將java選項–Xms=200m，還是有該錯誤。spooldir裡面的內容大約有1GB，數10個檔案。後面有人說memory channel 的capacity 應為 每秒傳輸event條數 * 要傳輸的時間秒數。然後capacity 應為agent1.channels.memch1.transactionCapactiy的10-100倍。我們剛開始將這兩個值設為 agent1.channels.memch1.capacity = 10000 agent1.channels.memch1.transactionCapactiy = 100 結果報了上述錯誤。後面將capacity設為1000000,該錯誤消失(報了outofmemory錯誤。因為我們記憶體設小了。)但程式可以正確執行。

如非註明轉載, 均為原創. 本站遵循知識共享CC協議,轉載請註明來源