獲取使用者地理位置

• 利用GeoIP對映使用者地理位置
• 直接處理日誌中的經緯度資訊

在Elasticsearch中，所有的資料都有一個型別，什麼樣的型別，就可以在其上做一些對應型別的特殊操作。geo資訊中的location欄位是經緯度，我們需要使用經緯度來定位地理位置；在Elasticsearch中，對於經緯度來說，要想使用Elasticsearch提供的地理位置查詢相關的功能，就需要構造一個結構，並且將其型別屬性設定為geo_point

利用GeoIP對映使用者地理位置

input {
        file {
                path => "/home/test/access_log"
 

                type => "access_log"
        }
}
filter {
        grok {
                match => {"message" => "%{COMBINEDAPACHELOG}"}
        }
        geoip {
                source => "clientip"     
                database => "/home/test/GeoLite2-City.mmdb"
                add_field => [ "[geoip][coordinates]"
 
, "%{[geoip][longitude]}" ]
                add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
                convert => [ "[geoip][coordinates]", "float"]
        }
}
output {
       stdout { codec => rubydebug }
       elasticsearch {
                hosts => [ "192.168.20.60:9200"
 
 ]
                index => "logstash-apache-access-%{+YYY.MM.dd}"
       }
}   

geoip外掛相關配置說明：

source:   必須設定 
database: 如果未指定，則預設為Logstash附帶的GeoLite2城市資料庫。
default_database_type: 預設值是city。可接受引數：City和ASN。
target: 預設值是“geoip”，指定Logstash應該儲存geoip資料的欄位。 例如，如果您有src_ip和dst_ip欄位，並希望兩個IP的GeoIP資訊，這可能是有用的。

預設預定義的模板必須只有匹配 logstash-* 的索引才會應用這個模板

kibana視覺化：

直接處理日誌中的經緯度資訊

日誌格式如下所示：

  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302411,863010030035393,39.2055473,117.1409302,10,66.1,1,12,2,9,1070,14.4,65,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302412,863010030039387,38.9934654,116.9013672,36,178.3,1,12,2,36,2628,14.1,93,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302413,863010030043751,39.2268219,117.0283508,36,152.6,1,12,2,37,769,14.2,69,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302412,863010030031244,,,0,-1.0,0,0,2,27,1113,14.4,87,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302413,863010030031269,38.9425621,116.9591980,48,71.7,1,12,2,48,1519,14.8,83,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302414,863010030039387,38.9932823,116.9013748,35,178.6,1,12,2,35,2626,14.1,91,0,0,0,0.0,0.0,

首先先通過grok,csv外掛獲取到相關地理位置資訊：

filter {
        grok {
                match => { "message" => "\[com\.test\.nlbs\.filter\.MsgDecodeFilter\] \- \SoutheastMessage\:(?<SoutheastMessa>(.*\n){0,}.*)" }
                remove_field => ["message"]
                add_field => {"head" =>"SoutheastMessage"}
        }
        csv {
                source => "SoutheastMessa"
                separator => ","
                columns => [ "#","time", "imei", "latitude", "longitude" ]
                remove_field => ["column6","column7","column8","column9","column10","column11","column12","column13","column14","column15","column16","column17","column18","column19","column20"]
        }
}

columns: 如果 columns未配置，或者沒有指定足夠的列，則預設列名稱為 “column1”， “column2”等。
remove_field: 移除指定的欄位

獲取資訊如下圖所示：

可以看出通過外掛我們過濾出了日誌中的經緯度資訊,但是是由 “longitude” 和 “latitude” 兩個欄位表示的。如果想要使用Elasticsearch提供的地理位置查詢相關的功能，就需要構造一個型別屬性為geo_point的結構：

filter {
        grok {
                match => { "message" => "\[com\.test\.nlbs\.filter\.MsgDecodeFilter\] \- \SoutheastMessage\:(?<SoutheastMessa>(.*\n){0,}.*)" }
                remove_field => ["message"]
                add_field => {"head" =>"SoutheastMessage"}
        }
        csv {
                source => "SoutheastMessa"
                separator => ","
                columns => [ "#","time", "imei", "latitude", "longitude" ]
                remove_field => ["column6","column7","column8","column9","column10","column11","column12","column13","column14","column15","column16","column17","column18","column19","column20"]
                add_field => [ "location", "%{longitude}" ]
                add_field => [ "location", "%{latitude}" ]
        }

        mutate {
                convert => [ "location", "float"]
        }
}

獲取的結果如下所示：

要構造一個型別屬性為geo_point的結構需要定義mapping：

PUT /myindex 
{
  "mappings": {
    "doc": { 
      "properties": { 
        "location":  { "type": "geo_point"  }
        }
      }
    }
}

檢視對映：

GET /myindex/_mapping
...
          "location": {
            "type": "geo_point"
          },
...

注意：這裡要有先後順序，先定義mapping

如果沒有定義mapping，預設的話會是“text”型別，在視覺化建立Coordinate Map時就會報“No Compatible Fields”錯誤：

最後使用高德地圖將地圖地名顯示成中文：
在kibana配置檔案中新增如下內容並重啟kibana服務：

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

最後成果如下：

參考