filebeat專題
一、filebeat概述
Filebeat是一個日誌文件托運工具，在你的服務器上安裝客戶端後，filebeat會監控日誌目錄或者指定的日誌文件，追蹤讀取這些文件（追蹤文件的變化，不停的讀），並且轉發這些信息到elasticsearch或者logstarsh中存放
工作流程
當你開啟filebeat程序的時候，它會啟動一個或多個探測器（prospectors）去檢測你指定的日誌目錄或文件，對於探測器找出的每一個日誌文件，filebeat啟動收割進程（harvester），每一個收割進程讀取一個日誌文件的新內容，並發送這些新的日誌數據到處理程序（spooler），處理程序會集合這些事件，最後filebeat會發送集合的數據到你指定的地點
二、filebeat的安裝
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.3.0-x86_64.rpm
[[email protected] ~]# rpm -ivh filebeat-5.3.0-x86_64.rpm
warning: filebeat-5.3.0-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [100%]
1:filebeat ########################################### [100%]
[[email protected] ~]# cd /etc/filebeat/
[[email protected] filebeat]# ll
total 80
-rw-r--r-- 1 root root 37050 Mar 23 11:23 filebeat.full.yml
-rw-r--r-- 1 root root 15641 Mar 23 11:23 filebeat.template-es2x.json
-rw-r--r-- 1 root root 12745 Mar 23 11:23 filebeat.template.json
-rw------- 1 root root 4196 Mar 23 11:23 filebeat.yml
[[email protected] filebeat]# mv filebeat.yml filebeat.yml.bak----------備份配置
三、配置filebeat
案例：收集服務器上nginx日誌和系統日誌
[[email protected] filebeat]# vim filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/nginx/access.log
tags: ["nginx-accesslog"]
document_type: nginxaccess
- input_type: log
paths:
- /var/log/messages
tags: ["sys-messages"]
document_type: sysmessages
tags: ["nginx-test-194"]
output.logstash:
hosts: ["10.2.11.222:5044"]
解釋：
input
1、path選項：filebeat發送給logstash的路徑，多個日誌可以使用*.log 通配
2、tags：會向log中添加一個標簽，該標簽可以提供給logstash用於區分不同客戶端不同業務的log
3、document_type：標記，跟tags差不多，區別不同的日誌來源
output：指定發送log到哪臺服務器上的那個服務，默認輸出到elasticsearch
[[email protected] filebeat]# service filebeat start
Starting filebeat: 2017/05/08 09:12:35.118324 beat.go:285: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/05/08 09:12:35.118355 beat.go:186: INFO Setup Beat: filebeat; Version: 5.3.0
2017/05/08 09:12:35.118506 metrics.go:23: INFO Metrics logging every 30s
2017/05/08 09:12:35.118589 logstash.go:90: INFO Max Retries set to: 3
2017/05/08 09:12:35.118690 outputs.go:108: INFO Activated logstash as output plugin.
2017/05/08 09:12:35.118863 publish.go:295: INFO Publisher name: blog
2017/05/08 09:12:35.119893 async.go:63: INFO Flush Interval set to: 1s
2017/05/08 09:12:35.119959 async.go:64: INFO Max Bulk Size set to: 2048
Config OK
[ OK ]
配置logstash收集日誌（在10.2.11.222）----stdout測試輸出
input {
beats {
port => 5044
}
}
output {
stdout {
codec => "rubydebug"
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f filebeat.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "08/May/2017:17:22:15 +0800|10.2.15.222|10.2.11.252|-|GET|/|HTTP/1.1|200|18|-|Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36|0.000|",
"@version" => "1",
"@timestamp" => "2017-05-08T09:22:25.193Z",
"input_type" => "log",
"beat" => {
"version" => "5.3.0",
"name" => "blog",
"hostname" => "blog"
},
"source" => "/var/log/nginx/access.log",
"type" => "nginxaccess",
"tags" => [
[0] "nginx-test-194",
[1] "nginx-accesslog",
[2] "beats_input_codec_plain_applied"
],
"offset" => 1094,
"host" => "blog"
}
[[email protected] ~]# cat file.conf
input {
beats {
port => 5044
}
}
filter {
if "nginx-accesslog" in [tags] {
grok {
match => { "message" => "%{HTTPDATE:timestamp}\|%{IP:remote_addr}\|%{IPORHOST:http_host}\|(?:%{DATA:http_x_forwarded_for}|-)\|%{DATA:request_method}\|%{DATA:request_uri}\|%{DATA:server_protocol}\|%{NUMBER:status}\|(?:%{NUMBER:body_bytes_sent}|-)\|(?:%{DATA:http_referer}|-)\|%{DATA:http_user_agent}\|(?:%{DATA:request_time}|-)\|"}
}
mutate {
convert => ["status","integer"]
convert => ["body_bytes_sent","integer"]
convert => ["request_time","float"]
}
geoip {
source=>"remote_addr"
}
date {
match => [ "timestamp","dd/MMM/YYYY:HH:mm:ss Z"]
}
useragent {
source=>"http_user_agent"
}
}
if "sys-messages" in [tags] {
grok {
match => { "message" => "%{SYSLOGLINE}" }
add_field => [ "received_at", "[email protected]}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "timestamp", "MMM d HH:mm:ss" ]
}
#ruby {
# code => "event[[email protected]] = event[[email protected]].getlocal"
#}
}
}
output {
stdout {
codec => "rubydebug"
}
}
測試
[[email protected] ~]# /opt/logstash/bin/logstash -f file.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "08/May/2017:17:35:04 +0800|10.2.15.222|10.2.11.252|-|GET|/|HTTP/1.1|200|18|-|Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36|0.000|",
"@version" => "1",
"@timestamp" => "2017-05-08T09:35:04.000Z",
"offset" => 1401,
"input_type" => "log",
"beat" => {
"name" => "blog",
"hostname" => "blog",
"version" => "5.3.0"
},
"type" => "nginxaccess",
"tags" => [
[0] "nginx-test-194",
[1] "nginx-accesslog",
[2] "beats_input_codec_plain_applied"
],
"source" => "/var/log/nginx/access.log",
"host" => "blog",
"timestamp" => "08/May/2017:17:35:04 +0800",
"remote_addr" => "10.2.15.222",
"http_host" => "10.2.11.252",
"http_x_forwarded_for" => "-",
"request_method" => "GET",
"request_uri" => "/",
"server_protocol" => "HTTP/1.1",
"status" => 200,
"body_bytes_sent" => 18,
"http_referer" => "-",
"http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36",
"request_time" => 0.0,
"name" => "Chrome",
"os" => "Windows 7",
"os_name" => "Windows 7",
"device" => "Other",
"major" => "57",
"minor" => "0",
"patch" => "2987"
}
測試輸出到elasticsearch上
output {
elasticsearch {
hosts => ["10.2.11.249:9200"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
document_type => "%{type}"
}
}
[[email protected] indices]# ll -----------查看elasticsearch服務上，出現該索引
total 56
drwxr-xr-x 8 elasticsearch elasticsearch 4096 May 8 17:42 logstash-nginxaccess-2017.05.08
kibana上查看
filebeat專題