Filebeat 原理詳解/配置檔案分析
-
配置檔案位置
對於rpm
和deb
,您將在以下位置找到配置檔案/etc/filebeat/filebeat.yml
。在Docker下,它位於/usr/share/filebeat/filebeat.yml
。對於mac
和win
以及zip文件
,請檢視剛剛提取的存檔。相同路徑下還有一個名為的完整示例配置檔案filebeat.reference.yml
,顯示了所有未棄用的配置選項。
-
讀取日誌配置:
-
專案日誌檔案
利用 Filebeat 去讀取檔案,paths 下面配置路徑地址。
/data/share/business_log/TA-*/debug.log
Filebeat 會自動去讀取business_log裡面的TA開頭的檔案。可以使用 Linux 的#=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations.
filebeat 對於多行日誌的處理,需要處理多行日誌的情況下
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ multiline: pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]' negate: true match: after
上面配置的意思是:不以時間格式開頭的行都合併到上一行的末尾(正則寫的不好,忽略忽略)
pattern
:正則表示式negate
:true 或 false;預設是false,匹配pattern的行合併到上一行;true,不匹配pattern的行合併到上一行match
:after 或 before,合併到上一行的末尾或開頭 還有更多兩個配置,預設也是註釋的,沒特殊要求可以不管它max_lines
: 500timeout
: 5smax_lines
:合併最大行,預設500timeout
:一次合併事件的超時時間,預設5s,防止合併消耗太多時間甚至卡死 -
nginx日誌檔案
#=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - type: log # Change to true to enable this prospector configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /usr/local/server/openresty/nginx/logs/access.log - /usr/local/server/openresty/nginx/logs/error.log #- /data/share/business_log/TA-*/debug.log #- c:\programdata\elasticsearch\logs\*
-
輸出配置
我們需要輸出到 Logstash 裡面,註釋掉 Elasticsearch 下面的配置項,並配置 Logstash 下面的配置,會將 Filebeat 讀取到的日誌檔案傳送到 hosts 裡面配置的 Logstash 伺服器上面去
#----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts # Logstash 不會組成叢集,但是 Filebeat 會自己去輪詢 Logstash 的伺服器,去找到可用的 Logstash 伺服器傳送過去 hosts: ["172.18.1.152:5044","172.18.1.153:5044","172.18.1.154:5044"] index: "logstash-%{+yyyy.MM.dd}" # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key"
Filebeat 啟動命令:nohup ./filebeat -e -c filebeat-TA.yml >/dev/null 2>&1 &
Filebeat 可以啟動多個,通過不同的 *-Filebeat.yml
配置檔案啟動