1. 程式人生 > >Centos下ELK環境搭建

Centos下ELK環境搭建

簡介:

 

ELK = Elasticsearch+Logstash+Kibana

Elasticsearch:儲存日誌,處理分析日誌,檢索日誌

Logstash:收集日誌

Kibana:web介面展示日誌

本來,Elasticsearch+Logstash+Kibana就可以完成一套日誌的是採集,處理+檢索,展示,但是Logstash是jvm實現的,耗得資源不少,部署到目標伺服器(產生日誌的應用伺服器)佔用資源多,故需比較輕量級的Filebeat部署到目標伺服器採集日誌

 

環境說明:

 

172.16.1.169 安裝 Elasticsearch(單機)

172.16.1.69   安裝Logstash

172.16.1.81   安裝Kibana

172.16.1.31   安裝Filebeat

當然,Elasticsearch+Logstash+Kibana+Filebeat四個傢伙同時部署到1臺伺服器也是可以的,只要伺服器效能吃得消(本人一臺2核8G伺服器同時部署四個傢伙啟動報記憶體溢位,跑不動),Elasticsearch配置成叢集的方式也是可以的,只要稍稍改下配置檔案

 

準備:

伺服器安裝jdk1.8以上

 

[root@iZwz98bpfv23id0ffnnis1Z ~]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
[root@iZwz98bpfv23id0ffnnis1Z ~]# javac -version
javac 1.8.0_181
[root@iZwz98bpfv23id0ffnnis1Z ~]# echo $JAVA_HOME
/usr/java/jdk1.8.0_181-amd64
[root@iZwz98bpfv23id0ffnnis1Z ~]#

 

下載檔案:

 

 到官網下載: https://www.elastic.co/cn/downloads

目前最新版本是7.6.2,所以本人安裝的也就是這個版本

由於官網下載比較慢,我這裡有個百度網盤的分享

連結:https://pan.baidu.com/s/1nWyaJJtB0Z3UAOuCNF055g
提取碼:t6qy


 

 安裝Elasticsearch:

新增elk使用者

 

[root@iZwz98bpfv23id0ffnnis1Z ~]# useradd elk
[root@iZwz98bpfv23id0ffnnis1Z ~]# passwd elk
Changing password for user elk.
New password:
BAD PASSWORD: The password contains the user name in some form
Retype new password:
passwd: all authentication tokens updated successfully.
[root@iZwz98bpfv23id0ffnnis1Z ~]#

 

開始安裝

 

[root@iZwz98bpfv23id0ffnnis1Z ~]# mv elasticsearch-7.6.2-linux-x86_64.tar.gz /home/elk/
[root@iZwz98bpfv23id0ffnnis1Z ~]# su elk
[elk@iZwz98bpfv23id0ffnnis1Z root]$ cd ~
[elk@iZwz98bpfv23id0ffnnis1Z ~]$ tar -zxvf elasticsearch-7.6.2-linux-x86_64.tar.gz
[elk@iZwz98bpfv23id0ffnnis1Z ~]$
[elk@iZwz98bpfv23id0ffnnis1Z ~]$ pwd
/home/elk
[elk@iZwz98bpfv23id0ffnnis1Z ~]$ mkdir -p var/es
[elk@iZwz98bpfv23id0ffnnis1Z ~]$ cd  elasticsearch-7.6.2/config/
[elk@iZwz98bpfv23id0ffnnis1Z config]$ cp elasticsearch.yml elasticsearch.yml_bak
[elk@iZwz98bpfv23id0ffnnis1Z config]$ vim elasticsearch.yml

 

elasticsearch.yml內容如下:

 

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /home/elk/var/es/data
#
# Path to log files:
#
path.logs: /home/elk/var/es/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: false
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 172.16.1.169
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ------------------------------------------------------------------------------
#
# 最後新增兩個配置。不然啟動報錯
bootstrap.system_call_filter: false
cluster.initial_master_nodes: ["node-1"]

  

啟動,發現報錯

 

[elk@iZwz98bpfv23id0ffnnis1Z config]$ cd ../
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ cd bin/
[elk@iZwz98bpfv23id0ffnnis1Z bin]$ ./elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_181-amd64/jre] does not meet this requirement
[2020-04-21T09:39:57,898][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.8gb], net total_space [39.2gb], types [rootfs]
[2020-04-21T09:39:57,903][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-04-21T09:39:57,949][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [dMk3D6nSRfSEM1y2LgOh8g], cluster name [my-application]
[2020-04-21T09:39:57,949][INFO ][o.e.n.Node               ] [node-1] version[7.6.2], pid[21937], build[default/tar/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-957.21.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2020-04-21T09:39:57,950][INFO ][o.e.n.Node               ] [node-1] JVM home [/usr/java/jdk1.8.0_181-amd64/jre]
[2020-04-21T09:39:57,950][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-2479772838674939679, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/elk/elasticsearch-7.6.2, -Des.path.conf=/home/elk/elasticsearch-7.6.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-04-21T09:40:00,886][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [flattened]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [frozen-indices]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [search-business-rules]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [spatial]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transform]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vectors]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-enrich]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2020-04-21T09:40:04,430][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/home/elk/elasticsearch-7.6.2/config/roles.yml]
[2020-04-21T09:40:05,254][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/22016] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-04-21T09:40:05,722][DEBUG][o.e.a.ActionModule       ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-04-21T09:40:05,837][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-04-21T09:40:06,640][INFO ][o.e.n.Node               ] [node-1] initialized
[2020-04-21T09:40:06,640][INFO ][o.e.n.Node               ] [node-1] starting ...
[2020-04-21T09:40:06,749][INFO ][o.e.t.TransportService   ] [node-1] publish_address {172.18.97.64:9300}, bound_addresses {0.0.0.0:9300}
[2020-04-21T09:40:07,022][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /home/elk/var/es/logs/my-application.log
[2020-04-21T09:40:07,028][INFO ][o.e.n.Node               ] [node-1] stopping ...
[2020-04-21T09:40:07,043][INFO ][o.e.n.Node               ] [node-1] stopped
[2020-04-21T09:40:07,043][INFO ][o.e.n.Node               ] [node-1] closing ...
[2020-04-21T09:40:07,062][INFO ][o.e.n.Node               ] [node-1] closed
[elk@iZwz98bpfv23id0ffnnis1Z bin]$

  

報錯如下:

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

 

切換到root使用者修改配置sysctl.conf
vi /etc/sysctl.conf
在尾行新增以下內容
vm.max_map_count=262144
並執行命令
sysctl -p

再次啟動:

 

[elk@iZwz98bpfv23id0ffnnis1Z bin]$ cd ../config/
[elk@iZwz98bpfv23id0ffnnis1Z config]$ clear
[elk@iZwz98bpfv23id0ffnnis1Z config]$ cd ../
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ cd bin/
[elk@iZwz98bpfv23id0ffnnis1Z bin]$ ./elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_181-amd64/jre] does not meet this requirement
[2020-04-21T10:21:39,286][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.8gb], net total_space [39.2gb], types [rootfs]
[2020-04-21T10:21:39,287][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-04-21T10:21:39,401][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [dMk3D6nSRfSEM1y2LgOh8g], cluster name [my-application]
[2020-04-21T10:21:39,402][INFO ][o.e.n.Node               ] [node-1] version[7.6.2], pid[24599], build[default/tar/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-957.21.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2020-04-21T10:21:39,402][INFO ][o.e.n.Node               ] [node-1] JVM home [/usr/java/jdk1.8.0_181-amd64/jre]
[2020-04-21T10:21:39,405][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-5400818995391898433, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/elk/elasticsearch-7.6.2, -Des.path.conf=/home/elk/elasticsearch-7.6.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-04-21T10:21:41,343][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2020-04-21T10:21:41,343][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2020-04-21T10:21:41,343][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [flattened]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [frozen-indices]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [search-business-rules]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [spatial]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transform]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vectors]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-enrich]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2020-04-21T10:21:45,493][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/home/elk/elasticsearch-7.6.2/config/roles.yml]
[2020-04-21T10:21:46,026][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/24678] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-04-21T10:21:46,539][DEBUG][o.e.a.ActionModule       ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-04-21T10:21:46,665][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-04-21T10:21:47,531][INFO ][o.e.n.Node               ] [node-1] initialized
[2020-04-21T10:21:47,531][INFO ][o.e.n.Node               ] [node-1] starting ...
[2020-04-21T10:21:47,655][INFO ][o.e.t.TransportService   ] [node-1] publish_address {172.18.97.64:9300}, bound_addresses {0.0.0.0:9300}
[2020-04-21T10:21:47,932][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-04-21T10:21:47,943][INFO ][o.e.c.c.Coordinator      ] [node-1] setting initial configuration to VotingConfiguration{dMk3D6nSRfSEM1y2LgOh8g}
[2020-04-21T10:21:48,106][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{dMk3D6nSRfSEM1y2LgOh8g}{LkwLqRLDRDudVsCsZ_upZw}{172.18.97.64}{172.18.97.64:9300}{dilm}{ml.machine_memory=3777253376, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node-1}{dMk3D6nSRfSEM1y2LgOh8g}{LkwLqRLDRDudVsCsZ_upZw}{172.18.97.64}{172.18.97.64:9300}{dilm}{ml.machine_memory=3777253376, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-04-21T10:21:48,161][INFO ][o.e.c.c.CoordinationState] [node-1] cluster UUID set to [QIDsWY3RTwOl86xiX0iH1Q]
[2020-04-21T10:21:48,187][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{dMk3D6nSRfSEM1y2LgOh8g}{LkwLqRLDRDudVsCsZ_upZw}{172.18.97.64}{172.18.97.64:9300}{dilm}{ml.machine_memory=3777253376, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-04-21T10:21:48,247][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {172.18.97.64:9200}, bound_addresses {0.0.0.0:9200}
[2020-04-21T10:21:48,247][INFO ][o.e.n.Node               ] [node-1] started
[2020-04-21T10:21:48,333][INFO ][o.e.g.GatewayService     ] [node-1] recovered [0] indices into cluster_state
[2020-04-21T10:21:48,610][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2020-04-21T10:21:48,651][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watches] for index patterns [.watches*]
[2020-04-21T10:21:48,704][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watch-history-10] for index patterns [.watcher-history-10*]
[2020-04-21T10:21:48,742][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [ilm-history] for index patterns [ilm-history-1*]
[2020-04-21T10:21:48,780][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.slm-history] for index patterns [.slm-history-1*]
[2020-04-21T10:21:48,826][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]
[2020-04-21T10:21:48,876][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2020-04-21T10:21:48,926][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2020-04-21T10:21:49,010][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2020-04-21T10:21:49,057][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2020-04-21T10:21:49,090][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-1] adding index lifecycle policy [watch-history-ilm-policy]
[2020-04-21T10:21:49,135][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-1] adding index lifecycle policy [ilm-history-ilm-policy]
[2020-04-21T10:21:49,180][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-1] adding index lifecycle policy [slm-history-ilm-policy]
[2020-04-21T10:21:49,295][INFO ][o.e.l.LicenseService     ] [node-1] license [406e5b20-653b-4621-84d8-d7e5135480a7] mode [basic] - valid
[2020-04-21T10:21:49,296][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled

 

好像是啟動成功了

瀏覽器訪問: http://172.16.1.169:9200

 

 

 啟動確認沒有問題了

 

[elk@iZwz98bpfv23id0ffnnis1Z bin]$ cd ../
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ vi starup.sh
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ chmod u+x starup.sh
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ps aux|grep elasticsearch-7.6.2
elk       7742  0.0  0.0 103344   888 pts/0    S+   10:31   0:00 grep elasticsearch-7.6.2
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ls
bin  config  data  jdk  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.asciidoc  replay_pid28957.log  starup.sh
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ./starup.sh
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_102/jre] does not meet this requirement
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ps aux|grep elasticsearch-7.6.2
elk       7809  185 15.6 3605960 1258136 pts/0 Sl   10:31   0:18 /usr/java/jdk1.8.0_102/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=COMPAT -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/elasticsearch-113030852514621270 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:MaxDirectMemorySize=536870912 -Des.path.home=/home/elk/elasticsearch-7.6.2 -Des.path.conf=/home/elk/elasticsearch-7.6.2/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/elk/elasticsearch-7.6.2/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
elk       7824  0.1  0.0  64124  4480 pts/0    Sl   10:31   0:00 /home/elk/elasticsearch-7.6.2/modules/x-pack-ml/platform/linux-x86_64/bin/controller
elk       7833  0.0  0.0 103344   888 pts/0    S+   10:31   0:00 grep elasticsearch-7.6.2
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$

  

starup.sh裡面的內容:
/home/elk/elasticsearch-7.6.2/bin/elasticsearch -d

安裝Logstash:

logstash可以用root使用者執行,logstash也是需要在jdk1.8以上版本執行,所以可以不用像elasticsearch那樣必須用新建一個elk使用者,本人直接了當,用root使用者,在root使用者根目錄安裝logstash

  

[root@jenkins ~]# ls
diamond  jdk-8u181-linux-x64.rpm  logstash-7.6.2.tar.gz  mysqld.log.gz
[root@jenkins ~]# tar -zxvf logstash-7.6.2.tar.gz
[root@jenkins ~]# cd logstash-7.6.2
[root@jenkins logstash-7.6.2]# vim config.conf
[root@jenkins logstash-7.6.2]# vim startup.sh
[root@jenkins logstash-7.6.2]# chmod u+x startup.sh

 

config.conf內容如下:

 

input {
    beats {
        port => "5044"
    }
}

 filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}

output {
    elasticsearch {
        hosts => [ "172.16.1.169:9200" ]
    }
}

  

startup.sh內容如下:

 

nohup /root/logstash-7.6.2/bin/logstash -f /root/logstash-7.6.2/config.conf &

 

啟動

 

[root@jenkins logstash-7.6.2]# ./startup.sh
[root@jenkins logstash-7.6.2]# nohup: appending output to `nohup.out'
[root@jenkins logstash-7.6.2]#
[root@jenkins logstash-7.6.2]# tail -f nohup.out
Sending Logstash logs to /root/logstash-7.6.2/logs which is now configured via log4j2.properties
[2020-04-21T11:30:38,888][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/root/logstash-7.6.2/data/queue"}
[2020-04-21T11:30:39,058][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/root/logstash-7.6.2/data/dead_letter_queue"}
[2020-04-21T11:30:39,480][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-21T11:30:39,489][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-21T11:30:39,516][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"5105c2e8-9563-4a47-9e9e-a8c037a04232", :path=>"/root/logstash-7.6.2/data/uuid"}
[2020-04-21T11:30:41,412][INFO ][org.reflections.Reflections] Reflections took 46 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-21T11:30:42,498][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.16.1.81:9200/]}}
[2020-04-21T11:30:42,734][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://172.16.1.81:9200/"}
[2020-04-21T11:30:42,808][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-21T11:30:42,817][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-21T11:30:42,948][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.16.1.81:9200"]}
[2020-04-21T11:30:43,030][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-04-21T11:30:43,156][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-04-21T11:30:43,171][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/root/logstash-7.6.2/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[2020-04-21T11:30:43,257][INFO ][logstash.outputs.elasticsearch][main] Creating rollover alias <logstash-{now/d}-000001>
[2020-04-21T11:30:43,360][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-04-21T11:30:43,369][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/root/logstash-7.6.2/config.conf"], :thread=>"#<Thread:0x3b9cb2d2 run>"}
[2020-04-21T11:30:43,574][INFO ][logstash.outputs.elasticsearch][main] Installing ILM policy {"policy"=>{"phases"=>{"hot"=>{"actions"=>{"rollover"=>{"max_size"=>"50gb", "max_age"=>"30d"}}}}}} to _ilm/policy/logstash-policy
[2020-04-21T11:30:44,405][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-04-21T11:30:44,460][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-04-21T11:30:44,523][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-21T11:30:44,583][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2020-04-21T11:30:44,937][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

 

至此,logstash安裝完畢

安裝Filebeat:

filebeat可以直接在root使用者執行

 

[root@redmine-dbs ~]# pwd
/root
[root@redmine-dbs ~]# tar -zxvf filebeat-7.6.2-linux-x86_64.tar.gz
[root@redmine-dbs ~]# mv filebeat-7.6.2-linux-x86_64 filebeat-7.6.2
[root@redmine-dbs ~]# cd filebeat-7.6.2
[root@redmine-dbs dbs filebeat-7.6.2]# vi config.yml
[root@redmine-dbs dbs filebeat-7.6.2]# vi startup.sh
[root@redmine-dbs dbs filebeat-7.6.2]# chmod u+x startup.sh

 

config.yml內容如下:

 

filebeat.inputs:
- type: log
  paths:
    -  /var/test.log
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:' 
  multiline.negate: false 
  multiline.match: after
output.logstash:
  hosts: ["172.16.1.69:5044"]

  starup.sh內容如下:

nohup /root/filebeat-7.6.2/filebeat -c /root/filebeat-7.6.2/config.yml &

  啟動:

[root@redmine-dbs filebeat-7.6.2]# ./startup.sh
[root@redmine-dbs filebeat-7.6.2]# nohup: appending output to `nohup.out'

[root@redmine-dbs filebeat-7.6.2]# tail -f logs/filebeat
2020-04-21T14:09:54.812+0800    WARN    beater/filebeat.go:152  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-04-21T14:09:54.813+0800    INFO    instance/beat.go:439    filebeat start running.
2020-04-21T14:09:54.813+0800    INFO    registrar/registrar.go:145      Loading registrar data from /root/filebeat-7.6.2/data/registry/filebeat/data.json
2020-04-21T14:09:54.813+0800    INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2020-04-21T14:09:54.813+0800    WARN    beater/filebeat.go:368  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-04-21T14:09:54.813+0800    INFO    crawler/crawler.go:72   Loading Inputs: 1
2020-04-21T14:09:54.813+0800    INFO    log/input.go:152        Configured paths: [/var/test.log]
2020-04-21T14:09:54.813+0800    INFO    input/input.go:114      Starting input of type: log; ID: 13852086122507548810
2020-04-21T14:09:54.813+0800    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2020-04-21T14:09:54.813+0800    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s

  

[root@redmine-dbs filebeat-7.6.2]# ps aux|grep filebeat-7.6.2
root      1422  0.0  1.4 260868 28608 pts/0    Sl   14:09   0:00 /root/filebeat-7.6.2/filebeat -c /root/filebeat-7.6.2/config.yml
root      1482  0.0  0.0 103244   872 pts/0    S+   14:11   0:00 grep filebeat-7.6.2
[root@redmine-dbs filebeat-7.6.2]# echo testlog > /var/test.log

 

安裝Kibana:

 

[root@localhost ~]# pwd
/root
[root@localhost ~]# tar -zxvf kibana-7.6.2-linux-x86_64.tar.gz
[root@localhost ~]# mv kibana-7.6.2-linux-x86_64 kibana-7.6.2
[root@localhost ~]# cd kibana-7.6.2
[root@localhost kibana-7.6.2]# cp config/kibana.yml config/kibana.yml_bak
[root@localhost kibana-7.6.2]# vi config/kibana.yml

  kibana.yml只開啟下面幾個配置項

server.port: 5601
server.host: "172.16.1.81"
elasticsearch.hosts: ["http://172.16.1.169:9200"]
kibana.index: ".kibana"

  啟動:

[root@localhost kibana-7.6.2]# ./bin/kibana --allow-root

  瀏覽器訪問:http://192.16.1.81:5601

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

此時,到172.16.1.31  vi /var/test.log  給這個日誌檔案隨便寫新增幾行,在上面就看到日誌記錄,並且可以檢索

至此,ALK環境搭建完畢

&n