filebeat logstash配置模板說明

為了讓不同類型的日誌記錄到不同index，實現日誌分類，需要更改默認的配置文件，ELK更新叠代速度很快，網上以前的文檔適用於之前的版本

filebeat的docment_type配置項已經在6版本中棄用，請使用本文配置

filebeat配置模板

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/messages
    - /var/log/secure
    - /var/log/*.log
  fields: 
    log_topics: syslog

-
 
 type: log
  enabled: true
  paths:
    - /data/www/logs/nginx_access.log
  fields: 
    log_topics: nginx_access

- type: log
  enabled: true
  paths:
    - /data/www/logs/nginx_error.log       ##nginx錯誤日誌位置
  fields: 
    log_topics: nginx_error               ##nginx錯誤日誌註明類型(logstash為不同類型創建不同索引)

- type: log
  enabled
 
: true
  paths:
    - /data/game/logs/dts_svc.log
    - /data/game/logs/dts_web.log
  fields: 
    log_topics: tomcat_svc_web
  multiline.pattern: '^\sINFO|^\sERROR|^\sDEBUG|^\sWARN'      ##將日誌info,error,debug,warn開頭的作為一行（用於java日誌多行合並,也可以用時間為開頭）
  multiline.negate: true
  multiline.match: after
  exclude_lines
 
: ['^ INFO','^ DEBUG']                         ##排除info,debug開頭的行
  include_lines: ["^ ERROR", "^ WARN"]                        ##將error,warn開頭的行傳給logstash

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:

output.logstash:
  hosts: ["localhost:5044"]

logstash配置模板

input {
    beats {
        port => 5044
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    }
}

output {
    if [fields][log_topics] == "nginx_access" {            ##按照type類型創建多個索引
        elasticsearch {
            hosts => ["192.168.1.146:9200"]
            index => "logstash-%{type}-%{+YYYY.MM.dd}_log"
            user => "elastic" 
            password => "QhQ0wS4G74JpKC7jRPJD"
        }

    }

    if [fields][log_topics] == "nginx_error" {            ##按照type類型創建多個索引
        elasticsearch {
            hosts => ["192.168.1.146:9200"]
            index => "logstash-%{[fields][log_topics]}-%{+YYYY.MM.dd}_log"
            user => "elastic" 
            password => "QhQ0wS4G74JpKC7jRPJD"
        }
    }

    if [fields][log_topics] == "tomcat_svc_web" {            ##按照type類型創建多個索引
        elasticsearch {
            hosts => ["192.168.1.146:9200"]
            index => "logstash-%{[fields][log_topics]}-%{+YYYY.MM.dd}_log"
            user => "elastic" 
            password => "QhQ0wS4G74JpKC7jRPJD"
        }
    }

    stdout { codec => rubydebug }
}

生成環境filebeat logstash配置模板