uri down rac ash 存儲 日誌格式 over byte install

一、Elasticserach安裝
1、Installation(elastic 6.3.2 版本 依賴 java JDK8)

下載合適的版本：
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
tar -xvf elasticsearch-6.4.0.tar.gz
cd elasticsearch-6.4.0/bin
./elasticsearch

2、集群健康檢查
curl -XGET http://127.0.0.1:9200/_cat/health?v 集群健康查看
curl -XGET http://127.0.0.1:9200/_cat/nodes?v 節點狀態查看

curl -XGET http://127.0.0.1:9200/_cat/indices?v 查看索引
curl -XPUT http://127.0.0.1:9200/customer?pretty 添加customer索引
curl -XDELETE http://127.0.0.1:9200/customer?pretty 刪除索引

3、Config
cluster.name: 集群名
node.name: 節點名
path.data: 數據存儲路徑
path.logs: 日誌存儲路徑
network.host: 監聽地址
http.port: 監聽端口
JVM 配置、logging配置 參考官方文檔
重要參數：

Path settings
Cluster name
Node name
Network host
Discovery settings
Heap size
Heap dump path
GC logging
Temp directory
重要的系統參數：

Disable swapping
Increase file descriptors
Ensure sufficient virtual memory
Ensure sufficient threads
JVM DNS cache settings

4、Running as daemon
./bin/elasticsearch -d -p pid 啟動
kill cat pid

停止

5、Set up X-Pack
參考x-pack破解方法

二、kibana安裝及配置

        wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz

tar -xzf kibana-6.4.0-linux-x86_64.tar.gz
cd kibana-6.4.0-linux-x86_64/

        1、啟動  ./bin/kibana   

        2、config：

server.port: 5601
server.host: "192.168.12.81"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "dinpay"

三、logstash安裝及配置
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz

配置 filebeat 收集Nginx日誌： （Nginx.conf）

input {
beats {
port => 5044
codec => "json"
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "test1-nginx-access-%{+YYYY.MM.dd}"
template_overwrite => true
user => elastic
password => dinpay
}
}

配置logstash.yml elastic安裝在本機（沒有配置https）

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: dinpay
xpack.monitoring.elasticsearch.url: ["http://127.0.0.1:9200"]

四、filebeat安裝及配置

wget https://artifacts.elastic.co/downloads/beats/filebeat/https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz

tar -xf filebeat-6.4.0-linux-x86_64.tar.gz

cd filebeat-6.4.0-linux-x86_64

配置 filebeat.yml

• type: log
  paths:

  • /var/log/nginx/access.log

output.logstash:

hosts: ["192.168.12.81:5044"]

xpack.monitoring:
enabled: true
elasticsearch:
hosts: ["http://192.168.12.81:9200", "http://192.168.12.81:9200"]
username: elastic
password: dinpay

五、Nginx 日誌格式配置：

log_format json ‘{"@timestamp":"$time_iso8601",‘
‘"host":"$server_addr",‘
‘"clientip":"$clientRealIp",‘
‘"remote_user":"$remote_user",‘
‘"request":"$request",‘
‘"http_user_agent":"$http_user_agent",‘
‘"size":$body_bytes_sent,‘
‘"responsetime":$request_time,‘
‘"upstreamtime":"$upstream_response_time",‘
‘"upstreamhost":"$upstream_addr",‘
‘"http_host":"$host",‘
‘"url":"$uri",‘
‘"domain":"$host",‘
‘"xff":"$http_x_forwarded_for",‘
‘"referer":"$http_referer",‘
‘"status":"$status",‘
‘"ss":"$upstream_status"}‘;

access_log  /var/log/nginx/access.log  json;

map $http_x_forwarded_for $clientRealIp {

""  $remote_addr;

~^(?P<firstAddr>[0-9\.]+),?.*$    $firstAddr;

}

Elastic+logstash+filebeat做Nginx日誌分析