使用docker安裝elk資料分析
阿新 • • 發佈:2019-01-02
docker:
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo sudo yum install docker-ce sudo systemctl start docker sudo docker run hello-world # docker中-d引數是在後臺執行, 若除錯的話, 可以改為--rm
Install Elasticsearch with Docker
sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:6.4.2 sys config /etc/sysctl.conf中新增: vm.max_map_count=262144 sudo grep vm.max_map_count /etc/sysctl.conf run: docker run -p 9200:9200 -p 9300:9300 -d --name elasticsearch -d -e "discovery.type=single-node" -e network.publish_host=0.0.0.0 docker.elastic.co/elasticsearch/elasticsearch:6.4.2 inspect status of cluster: curl http://127.0.0.1:9200/_cat/health 檢視索引: curl -X GET 'http://localhost:9200/_cat/indices?v' 檢視type: curl 'localhost:9200/_mapping?pretty=true' 新建 Index: curl -X PUT 'localhost:9200/weather' 查詢記錄: curl 'localhost:9200/accounts/person/_search' # /Index/Type/_search
Kibana
docker pull docker.elastic.co/kibana/kibana:6.4.2 docker run -p 5601:5601 --name kibana -d --link elasticsearch -e ELASTICSEARCH_URL=http://10.97.88.71:9200 -e elasticsearch.ssl.verify=false -e server.host=0.0.0.0 docker.elastic.co/kibana/kibana:6.4.2 # elasticsearch 為 Elasticsearch 的docker name # 10.97.88.71, 使用docker的時候, 不能用localhost http://10.97.88.71:5601
Logstash
docker pull docker.elastic.co/logstash/logstash:6.4.2
docker run --name logstash --rm -p 5144:5144 --link elasticsearch -e xpack.monitoring.enabled=true -e ELASTICSEARCH_URL=http://10.97.88.71:9200 -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:6.4.2
配置檔案 logstash.yml
input{
syslog {
type => "icc_rpc_log"
port => "5144"
}
}
filter {
if [type] == "icc_rpc_log"{
grok {
patterns_dir => "/usr/local/logstash/patterns" //設定自定義正則路徑
# match => { "message" => "%{IP:client_id_address} %{LOGLEVEL:loglevel}" }
match => { "message" => "%{TIMESTAMP_ISO8601:log_date} %{LOGLEVEL:loglevel}\:index code %{NUMBER:index_code} is invoked by %{IP:client_id_address}\(%{DATA:user_cd}\-%{DATA:user_name}\) with parameter_list %{DATA:parameter_list} \+ default_parameter %{GREEDYDATA:parameter_default}" }
}
}
}
output {
if [type] == "icc_rpc_log" and [loglevel] == "INFO"{
stdout { codec => rubydebug }
}
if [type] == "icc_rpc_log" and [loglevel] == "ERROR"{
elasticsearch {
hosts => ["10.97.88.71:9200"]
index => "icc_calc_log"
# index => "system-syslog-log-%{+YYYY.MM.dd}"
}
}
}
測試資料:
2018-10-19 08:50:47 INFO:index code 000000001 is invoked by 1.2.2.2(-Anonymous) with parameter_list [{'asset_code': '000001', 'benm_code': '000002', 'yield_date_type': None, 'yield_type': None}] + default_parameter {'end_date': '2017-05-01', 'start_date': '2017-05-01', 'freq_code': 'D', 'riskfree_benm_code': '000003', 'annual_flag': False}
其他相關站點:
Logstash 最佳實踐: https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/output/elasticsearch.html
Kibana(一張圖片勝過千萬行日誌): https://www.cnblogs.com/cjsblog/p/9476813.html
kibana visualize新增自定義查詢: https://blog.csdn.net/xr568897472/article/details/71540937
全文搜尋引擎 Elasticsearch 入門教程: http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html
使用Docker搭建ELK日誌系統: http://chenzhijun.me/2017/12/27/elk-docker/
logstash-patterns-core: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
1分鐘搭建Elasticsearch視覺化(沒試過): https://blog.csdn.net/dounine/article/details/78887792