首先我們用的是elasticsearch+kibana+logstash+filebeat

客戶端filebeat收集日誌後經過服務端logstash規則處理後儲存到elasticsearch中，在kibana中展示。

以nginx日誌為例

1.我遇到的問題是，logstash中filter的規則似乎未生效，kibana中新建索引總是沒有geoip引數

logstash配置檔案如下

input {
beats{
port => 5044
codec => json {
charset => "UTF-8"
}
}
}

filter{
grok {
match => {"message" => '%{DATA:http_x_forwarded_for} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:request_uri}"%{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{DATA:http_referer} "%{DATA:http_user_agent}"'}
}
if "63nginx_access" in [tags] {
json{
source => "message"
}
if [user_ua] != "-" {
useragent {
target => "agent" #agent將過來出的user agent的資訊配置到了單獨的欄位中
source => "user_ua" #這個表示對message裡面的哪個欄位進行分析
}
}
if [http_x_forwarded_for] != "-" {
geoip {
source => "http_x_forwarded_for"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
}
}

output {
if[type] == "63nginx_access"{
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash_63nginx_access.%{+YYYY.MM.dd}"
}
}

1.1 建立logstash測試檔案用來除錯  vim logstash.test.conf

input {
stdin {}
}

filter {
grok {
match => {"message" => '%{DATA:http_x_forwarded_for} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:request_uri}"%{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{DATA:http_referer} "%{DATA:http_user_agent}"'}
}

if [http_x_forwarded_for] != '-'{
geoip {
source => "http_x_forwarded_for"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}

}

output {
stdout {
codec => rubydebug
}
}

啟動logstash

./bin/logstash -f   logstash.test.conf

啟動後貼上一行nginx的日誌

geoip為空，因為我們nginx的http_x_forwarded_for獲取到兩個ip,接著我用單ip測試，一定要是公網ip(內網ip在規則中被過濾了)

啟動logstash

./bin/logstash -f   logstash.test.conf

輸入

211.154.222.21 - - [26/Oct/2018:15:07:20 +0800] "GET /pp/index.php?/categories/posted-monthly-list-any-any/start-111210 HTTP/1.0"200 21761  "-""Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"

顯然這樣就獲取到geoip的資訊了，接著需要調整下nginx日誌了