1. 程式人生 > >Flume監控指標項

Flume監控指標項

配置監控

  1. 修改flume.env.sh

    export JAVA_OPTS=”-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false”
  2. 使用以下指令碼

    flume-ng agent \
    --name a1 \
    --conf $FLUME_HOME/conf \
    --conf-file $FLUME_HOME/conf/exec-memory-hdfs-partition.conf \
    -Dflume.root.logger=INFO,console \
    -Dflume.monitoring.type=http \
    -Dflume.monitoring.port=1234
    

指標項整理

OS監控項

objectName 指標項 說明
java.lang:type=OperatingSystem FreePhysicalMemorySize 空閒實體記憶體
java.lang:type=OperatingSystem SystemCpuLoad 系統CPU利用率
java.lang:type=OperatingSystem ProcessCpuLoad 程序CPU利用率
java.lang:type=GarbageCollector,name=PS Scavenge CollectionCount GC次數

json資料格式

{
    "SOURCE.src-1":{
        "OpenConnectionCount":"0",
        "Type":"SOURCE",
        "AppendBatchAcceptedCount":"1355",
        "AppendBatchReceivedCount":"1355",
        "EventAcceptedCount":"28286",
        "AppendReceivedCount":"0",
        "StopTime":"0",
        "StartTime":"1442566410435",
        "EventReceivedCount":"28286",
        "AppendAcceptedCount":"0"
    },
    "CHANNEL.ch-1":{
        "EventPutSuccessCount":"28286",
        "ChannelFillPercentage":"0.0",
        "Type":"CHANNEL",
        "StopTime":"0",
        "EventPutAttemptCount":"28286",
        "ChannelSize":"0",
        "StartTime":"1442566410326",
        "EventTakeSuccessCount":"28286",
        "ChannelCapacity":"1000000",
        "EventTakeAttemptCount":"313734329512"
    },
    "SINK.sink-1":{
        "Type":"SINK",
        "ConnectionClosedCount":"0",
        "EventDrainSuccessCount":"28286",
        "KafkaEventSendTimer":"482493",
        "BatchCompleteCount":"0",
        "ConnectionFailedCount":"0",
        "EventDrainAttemptCount":"0",
        "ConnectionCreatedCount":"0",
        "BatchEmptyCount":"0",
        "StopTime":"0",
        "RollbackCount":"9",
        "StartTime":"1442566411897",
        "BatchUnderflowCount":"0"
    }
}

指標項說明

source監控項

objectName
(會隨實際情況而變化)
指標項 說明
org.apache.flume.source:type=r1 OpenConnectionCount 目前與客戶端或sink保持連線的總數量
org.apache.flume.source:type=r1 AppendBatchAcceptedCount 成功提交到channel的批次的總數量
org.apache.flume.source:type=r1 AppendBatchReceivedCount 接收到事件批次的總數量
org.apache.flume.source:type=r1 AppendAcceptedCount 逐條錄入的次數
org.apache.flume.source:type=r1 AppendReceivedCount 每批只有一個事件的事件總數量
org.apache.flume.source:type=r1 EventAcceptedCount 成功寫出到channel的事件總數量
org.apache.flume.source:type=r1 EventReceivedCount 目前為止source已經接收到的事件總數量
org.apache.flume.source:type=r1 StartTime source啟動時的毫秒值時間
org.apache.flume.source:type=r1 StopTime source停止時的毫秒值時間,為0表示一直在執行

channel監控項

objectName
(會隨實際情況而變化)
指標項 說明
org.apache.flume.channel:type=c1 EventPutAttemptCount Source嘗試寫入Channe的事件總次數
org.apache.flume.channel:type=c1 EventPutSuccessCount 成功寫入channel且提交的事件總次數
org.apache.flume.channel:type=c1 EventTakeAttemptCount sink嘗試從channel拉取事件的總次數。
org.apache.flume.channel:type=c1 EventTakeSuccessCount sink成功從channel讀取事件的總數量
org.apache.flume.channel:type=c1 ChannelSize 目前channel中事件的總數量
org.apache.flume.channel:type=c1 ChannelCapacity channel的容量
org.apache.flume.channel:type=c1 ChannelFillPercentage channel已填入的百分比
org.apache.flume.channel:type=c1 StartTime channel啟動時的毫秒值時間
org.apache.flume.channel:type=c1 StopTime channel停止時的毫秒值時間,為0表示一直在執行

sink監控項

objectName
(會隨實際情況而變化)
指標項 說明
org.apache.flume.sink:type=k1 ConnectionCreatedCount 建立的連線數量
org.apache.flume.sink:type=k1 ConnectionClosedCount 關閉的連線數量
org.apache.flume.sink:type=k1 ConnectionFailedCount 由於錯誤關閉的連線數量
org.apache.flume.sink:type=k1 BatchEmptyCount 批量處理event的個數為0的數量-表示source寫入資料的速度比sink處理資料的速度慢
org.apache.flume.sink:type=k1 BatchUnderflowCount 批量處理event的個數小於批處理大小的數量
org.apache.flume.sink:type=k1 BatchCompleteCount 批量處理event的個數等於批處理大小的數量
org.apache.flume.sink:type=k1 EventDrainAttemptCount sink嘗試寫出到儲存的事件總數量
org.apache.flume.sink:type=k1 EventDrainSuccessCount sink成功寫出到儲存的事件總數量
org.apache.flume.sink:type=k1 StartTime channel啟動時的毫秒值時間
org.apache.flume.sink:type=k1 StopTime channel停止時的毫秒值時間,為0表示一直在執行