Centos下ELK環境搭建
阿新 • • 發佈：2020-04-21
簡介：
ELK = Elasticsearch+Logstash+Kibana
Elasticsearch：儲存日誌，處理分析日誌，檢索日誌
Logstash：收集日誌
Kibana：web介面展示日誌
本來，Elasticsearch+Logstash+Kibana就可以完成一套日誌的是採集，處理+檢索，展示，但是Logstash是jvm實現的，耗得資源不少，部署到目標伺服器（產生日誌的應用伺服器）佔用資源多，故需比較輕量級的Filebeat部署到目標伺服器採集日誌
環境說明：
172.16.1.169 安裝 Elasticsearch（單機）
172.16.1.69 安裝Logstash
172.16.1.81 安裝Kibana
172.16.1.31 安裝Filebeat
當然，Elasticsearch+Logstash+Kibana+Filebeat四個傢伙同時部署到1臺伺服器也是可以的，只要伺服器效能吃得消（本人一臺2核8G伺服器同時部署四個傢伙啟動報記憶體溢位，跑不動），Elasticsearch配置成叢集的方式也是可以的，只要稍稍改下配置檔案
準備:
伺服器安裝jdk1.8以上
[root@iZwz98bpfv23id0ffnnis1Z ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) [root@iZwz98bpfv23id0ffnnis1Z ~]# javac -version javac 1.8.0_181 [root@iZwz98bpfv23id0ffnnis1Z ~]# echo $JAVA_HOME /usr/java/jdk1.8.0_181-amd64 [root@iZwz98bpfv23id0ffnnis1Z ~]#
下載檔案：
到官網下載： https://www.elastic.co/cn/downloads
目前最新版本是7.6.2，所以本人安裝的也就是這個版本
由於官網下載比較慢，我這裡有個百度網盤的分享
連結：https://pan.baidu.com/s/1nWyaJJtB0Z3UAOuCNF055g
提取碼：t6qy
安裝Elasticsearch：
新增elk使用者
[root@iZwz98bpfv23id0ffnnis1Z ~]# useradd elk [root@iZwz98bpfv23id0ffnnis1Z ~]# passwd elk Changing password for user elk. New password: BAD PASSWORD: The password contains the user name in some form Retype new password: passwd: all authentication tokens updated successfully. [root@iZwz98bpfv23id0ffnnis1Z ~]#
開始安裝
[root@iZwz98bpfv23id0ffnnis1Z ~]# mv elasticsearch-7.6.2-linux-x86_64.tar.gz /home/elk/ [root@iZwz98bpfv23id0ffnnis1Z ~]# su elk [elk@iZwz98bpfv23id0ffnnis1Z root]$ cd ~ [elk@iZwz98bpfv23id0ffnnis1Z ~]$ tar -zxvf elasticsearch-7.6.2-linux-x86_64.tar.gz [elk@iZwz98bpfv23id0ffnnis1Z ~]$ [elk@iZwz98bpfv23id0ffnnis1Z ~]$ pwd /home/elk [elk@iZwz98bpfv23id0ffnnis1Z ~]$ mkdir -p var/es [elk@iZwz98bpfv23id0ffnnis1Z ~]$ cd elasticsearch-7.6.2/config/ [elk@iZwz98bpfv23id0ffnnis1Z config]$ cp elasticsearch.yml elasticsearch.yml_bak [elk@iZwz98bpfv23id0ffnnis1Z config]$ vim elasticsearch.yml
elasticsearch.yml內容如下：
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: my-application # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /home/elk/var/es/data # # Path to log files: # path.logs: /home/elk/var/es/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # bootstrap.memory_lock: false # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 172.16.1.169 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true # # ------------------------------------------------------------------------------ # # 最後新增兩個配置。不然啟動報錯 bootstrap.system_call_filter: false cluster.initial_master_nodes: ["node-1"]
啟動，發現報錯
[elk@iZwz98bpfv23id0ffnnis1Z config]$ cd ../
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ cd bin/
[elk@iZwz98bpfv23id0ffnnis1Z bin]$ ./elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_181-amd64/jre] does not meet this requirement
[2020-04-21T09:39:57,898][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.8gb], net total_space [39.2gb], types [rootfs]
[2020-04-21T09:39:57,903][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-04-21T09:39:57,949][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [dMk3D6nSRfSEM1y2LgOh8g], cluster name [my-application]
[2020-04-21T09:39:57,949][INFO ][o.e.n.Node ] [node-1] version[7.6.2], pid[21937], build[default/tar/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-957.21.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2020-04-21T09:39:57,950][INFO ][o.e.n.Node ] [node-1] JVM home [/usr/java/jdk1.8.0_181-amd64/jre]
[2020-04-21T09:39:57,950][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-2479772838674939679, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/elk/elasticsearch-7.6.2, -Des.path.conf=/home/elk/elasticsearch-7.6.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-04-21T09:40:00,886][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService ] [node-1] loaded module [analysis-common]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService ] [node-1] loaded module [flattened]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService ] [node-1] loaded module [frozen-indices]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2020-04-21T09:40:00,887][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [rank-eval]
[2020-04-21T09:40:00,888][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [repository-url]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [search-business-rules]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [spatial]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transform]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [vectors]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-analytics]
[2020-04-21T09:40:00,889][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ccr]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-core]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-deprecation]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-enrich]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-graph]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ilm]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-logstash]
[2020-04-21T09:40:00,890][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ml]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-monitoring]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-rollup]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-security]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-sql]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-voting-only-node]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-watcher]
[2020-04-21T09:40:00,891][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2020-04-21T09:40:04,430][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/home/elk/elasticsearch-7.6.2/config/roles.yml]
[2020-04-21T09:40:05,254][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/22016] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-04-21T09:40:05,722][DEBUG][o.e.a.ActionModule ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-04-21T09:40:05,837][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-04-21T09:40:06,640][INFO ][o.e.n.Node ] [node-1] initialized
[2020-04-21T09:40:06,640][INFO ][o.e.n.Node ] [node-1] starting ...
[2020-04-21T09:40:06,749][INFO ][o.e.t.TransportService ] [node-1] publish_address {172.18.97.64:9300}, bound_addresses {0.0.0.0:9300}
[2020-04-21T09:40:07,022][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /home/elk/var/es/logs/my-application.log
[2020-04-21T09:40:07,028][INFO ][o.e.n.Node ] [node-1] stopping ...
[2020-04-21T09:40:07,043][INFO ][o.e.n.Node ] [node-1] stopped
[2020-04-21T09:40:07,043][INFO ][o.e.n.Node ] [node-1] closing ...
[2020-04-21T09:40:07,062][INFO ][o.e.n.Node ] [node-1] closed
[elk@iZwz98bpfv23id0ffnnis1Z bin]$
報錯如下：
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
切換到root使用者修改配置sysctl.conf
vi /etc/sysctl.conf
在尾行新增以下內容
vm.max_map_count=262144
並執行命令
sysctl -p
再次啟動：
[elk@iZwz98bpfv23id0ffnnis1Z bin]$ cd ../config/
[elk@iZwz98bpfv23id0ffnnis1Z config]$ clear
[elk@iZwz98bpfv23id0ffnnis1Z config]$ cd ../
[elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ cd bin/
[elk@iZwz98bpfv23id0ffnnis1Z bin]$ ./elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_181-amd64/jre] does not meet this requirement
[2020-04-21T10:21:39,286][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.8gb], net total_space [39.2gb], types [rootfs]
[2020-04-21T10:21:39,287][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-04-21T10:21:39,401][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [dMk3D6nSRfSEM1y2LgOh8g], cluster name [my-application]
[2020-04-21T10:21:39,402][INFO ][o.e.n.Node ] [node-1] version[7.6.2], pid[24599], build[default/tar/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-957.21.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2020-04-21T10:21:39,402][INFO ][o.e.n.Node ] [node-1] JVM home [/usr/java/jdk1.8.0_181-amd64/jre]
[2020-04-21T10:21:39,405][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-5400818995391898433, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/elk/elasticsearch-7.6.2, -Des.path.conf=/home/elk/elasticsearch-7.6.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-04-21T10:21:41,343][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2020-04-21T10:21:41,343][INFO ][o.e.p.PluginsService ] [node-1] loaded module [analysis-common]
[2020-04-21T10:21:41,343][INFO ][o.e.p.PluginsService ] [node-1] loaded module [flattened]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [frozen-indices]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2020-04-21T10:21:41,344][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [rank-eval]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [repository-url]
[2020-04-21T10:21:41,345][INFO ][o.e.p.PluginsService ] [node-1] loaded module [search-business-rules]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [spatial]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transform]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [vectors]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-analytics]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ccr]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-core]
[2020-04-21T10:21:41,346][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-deprecation]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-enrich]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-graph]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ilm]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-logstash]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ml]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-monitoring]
[2020-04-21T10:21:41,347][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-rollup]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-security]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-sql]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-voting-only-node]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-watcher]
[2020-04-21T10:21:41,348][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2020-04-21T10:21:45,493][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/home/elk/elasticsearch-7.6.2/config/roles.yml]
[2020-04-21T10:21:46,026][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/24678] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-04-21T10:21:46,539][DEBUG][o.e.a.ActionModule ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-04-21T10:21:46,665][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-04-21T10:21:47,531][INFO ][o.e.n.Node ] [node-1] initialized
[2020-04-21T10:21:47,531][INFO ][o.e.n.Node ] [node-1] starting ...
[2020-04-21T10:21:47,655][INFO ][o.e.t.TransportService ] [node-1] publish_address {172.18.97.64:9300}, bound_addresses {0.0.0.0:9300}
[2020-04-21T10:21:47,932][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-04-21T10:21:47,943][INFO ][o.e.c.c.Coordinator ] [node-1] setting initial configuration to VotingConfiguration{dMk3D6nSRfSEM1y2LgOh8g}
[2020-04-21T10:21:48,106][INFO ][o.e.c.s.MasterService ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{dMk3D6nSRfSEM1y2LgOh8g}{LkwLqRLDRDudVsCsZ_upZw}{172.18.97.64}{172.18.97.64:9300}{dilm}{ml.machine_memory=3777253376, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node-1}{dMk3D6nSRfSEM1y2LgOh8g}{LkwLqRLDRDudVsCsZ_upZw}{172.18.97.64}{172.18.97.64:9300}{dilm}{ml.machine_memory=3777253376, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-04-21T10:21:48,161][INFO ][o.e.c.c.CoordinationState] [node-1] cluster UUID set to [QIDsWY3RTwOl86xiX0iH1Q]
[2020-04-21T10:21:48,187][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{dMk3D6nSRfSEM1y2LgOh8g}{LkwLqRLDRDudVsCsZ_upZw}{172.18.97.64}{172.18.97.64:9300}{dilm}{ml.machine_memory=3777253376, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-04-21T10:21:48,247][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {172.18.97.64:9200}, bound_addresses {0.0.0.0:9200}
[2020-04-21T10:21:48,247][INFO ][o.e.n.Node ] [node-1] started
[2020-04-21T10:21:48,333][INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state
[2020-04-21T10:21:48,610][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2020-04-21T10:21:48,651][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watches] for index patterns [.watches*]
[2020-04-21T10:21:48,704][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watch-history-10] for index patterns [.watcher-history-10*]
[2020-04-21T10:21:48,742][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [ilm-history] for index patterns [ilm-history-1*]
[2020-04-21T10:21:48,780][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.slm-history] for index patterns [.slm-history-1*]
[2020-04-21T10:21:48,826][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]
[2020-04-21T10:21:48,876][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2020-04-21T10:21:48,926][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2020-04-21T10:21:49,010][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2020-04-21T10:21:49,057][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2020-04-21T10:21:49,090][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-1] adding index lifecycle policy [watch-history-ilm-policy]
[2020-04-21T10:21:49,135][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-1] adding index lifecycle policy [ilm-history-ilm-policy]
[2020-04-21T10:21:49,180][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-1] adding index lifecycle policy [slm-history-ilm-policy]
[2020-04-21T10:21:49,295][INFO ][o.e.l.LicenseService ] [node-1] license [406e5b20-653b-4621-84d8-d7e5135480a7] mode [basic] - valid
[2020-04-21T10:21:49,296][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled
好像是啟動成功了
瀏覽器訪問： http://172.16.1.169:9200
啟動確認沒有問題了
[elk@iZwz98bpfv23id0ffnnis1Z bin]$ cd ../ [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ vi starup.sh [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ chmod u+x starup.sh [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ps aux|grep elasticsearch-7.6.2 elk 7742 0.0 0.0 103344 888 pts/0 S+ 10:31 0:00 grep elasticsearch-7.6.2 [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ls bin config data jdk lib LICENSE.txt logs modules NOTICE.txt plugins README.asciidoc replay_pid28957.log starup.sh [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ./starup.sh future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_102/jre] does not meet this requirement [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$ ps aux|grep elasticsearch-7.6.2 elk 7809 185 15.6 3605960 1258136 pts/0 Sl 10:31 0:18 /usr/java/jdk1.8.0_102/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=COMPAT -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/elasticsearch-113030852514621270 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:MaxDirectMemorySize=536870912 -Des.path.home=/home/elk/elasticsearch-7.6.2 -Des.path.conf=/home/elk/elasticsearch-7.6.2/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/elk/elasticsearch-7.6.2/lib/* org.elasticsearch.bootstrap.Elasticsearch -d elk 7824 0.1 0.0 64124 4480 pts/0 Sl 10:31 0:00 /home/elk/elasticsearch-7.6.2/modules/x-pack-ml/platform/linux-x86_64/bin/controller elk 7833 0.0 0.0 103344 888 pts/0 S+ 10:31 0:00 grep elasticsearch-7.6.2 [elk@iZwz98bpfv23id0ffnnis1Z elasticsearch-7.6.2]$
starup.sh裡面的內容：
/home/elk/elasticsearch-7.6.2/bin/elasticsearch -d
安裝Logstash：
logstash可以用root使用者執行，logstash也是需要在jdk1.8以上版本執行，所以可以不用像elasticsearch那樣必須用新建一個elk使用者，本人直接了當，用root使用者，在root使用者根目錄安裝logstash
[root@jenkins ~]# ls diamond jdk-8u181-linux-x64.rpm logstash-7.6.2.tar.gz mysqld.log.gz [root@jenkins ~]# tar -zxvf logstash-7.6.2.tar.gz [root@jenkins ~]# cd logstash-7.6.2 [root@jenkins logstash-7.6.2]# vim config.conf [root@jenkins logstash-7.6.2]# vim startup.sh [root@jenkins logstash-7.6.2]# chmod u+x startup.sh
config.conf內容如下：
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "172.16.1.169:9200" ]
}
}
startup.sh內容如下：
nohup /root/logstash-7.6.2/bin/logstash -f /root/logstash-7.6.2/config.conf &
啟動
[root@jenkins logstash-7.6.2]# ./startup.sh
[root@jenkins logstash-7.6.2]# nohup: appending output to `nohup.out'
[root@jenkins logstash-7.6.2]#
[root@jenkins logstash-7.6.2]# tail -f nohup.out
Sending Logstash logs to /root/logstash-7.6.2/logs which is now configured via log4j2.properties
[2020-04-21T11:30:38,888][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/root/logstash-7.6.2/data/queue"}
[2020-04-21T11:30:39,058][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/root/logstash-7.6.2/data/dead_letter_queue"}
[2020-04-21T11:30:39,480][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-21T11:30:39,489][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-21T11:30:39,516][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"5105c2e8-9563-4a47-9e9e-a8c037a04232", :path=>"/root/logstash-7.6.2/data/uuid"}
[2020-04-21T11:30:41,412][INFO ][org.reflections.Reflections] Reflections took 46 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-21T11:30:42,498][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.16.1.81:9200/]}}
[2020-04-21T11:30:42,734][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://172.16.1.81:9200/"}
[2020-04-21T11:30:42,808][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-21T11:30:42,817][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-21T11:30:42,948][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.16.1.81:9200"]}
[2020-04-21T11:30:43,030][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-04-21T11:30:43,156][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-04-21T11:30:43,171][INFO ][logstash.filters.geoip ][main] Using geoip database {:path=>"/root/logstash-7.6.2/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[2020-04-21T11:30:43,257][INFO ][logstash.outputs.elasticsearch][main] Creating rollover alias <logstash-{now/d}-000001>
[2020-04-21T11:30:43,360][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-04-21T11:30:43,369][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/root/logstash-7.6.2/config.conf"], :thread=>"#<Thread:0x3b9cb2d2 run>"}
[2020-04-21T11:30:43,574][INFO ][logstash.outputs.elasticsearch][main] Installing ILM policy {"policy"=>{"phases"=>{"hot"=>{"actions"=>{"rollover"=>{"max_size"=>"50gb", "max_age"=>"30d"}}}}}} to _ilm/policy/logstash-policy
[2020-04-21T11:30:44,405][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-04-21T11:30:44,460][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-04-21T11:30:44,523][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-21T11:30:44,583][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2020-04-21T11:30:44,937][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
至此，logstash安裝完畢
安裝Filebeat：
filebeat可以直接在root使用者執行
[root@redmine-dbs ~]# pwd /root [root@redmine-dbs ~]# tar -zxvf filebeat-7.6.2-linux-x86_64.tar.gz [root@redmine-dbs ~]# mv filebeat-7.6.2-linux-x86_64 filebeat-7.6.2 [root@redmine-dbs ~]# cd filebeat-7.6.2 [root@redmine-dbs dbs filebeat-7.6.2]# vi config.yml [root@redmine-dbs dbs filebeat-7.6.2]# vi startup.sh [root@redmine-dbs dbs filebeat-7.6.2]# chmod u+x startup.sh
config.yml內容如下：
filebeat.inputs:
- type: log
paths:
- /var/test.log
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
output.logstash:
hosts: ["172.16.1.69:5044"]
starup.sh內容如下：
nohup /root/filebeat-7.6.2/filebeat -c /root/filebeat-7.6.2/config.yml &
啟動：
[root@redmine-dbs filebeat-7.6.2]# ./startup.sh [root@redmine-dbs filebeat-7.6.2]# nohup: appending output to `nohup.out' [root@redmine-dbs filebeat-7.6.2]# tail -f logs/filebeat 2020-04-21T14:09:54.812+0800 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning. 2020-04-21T14:09:54.813+0800 INFO instance/beat.go:439 filebeat start running. 2020-04-21T14:09:54.813+0800 INFO registrar/registrar.go:145 Loading registrar data from /root/filebeat-7.6.2/data/registry/filebeat/data.json 2020-04-21T14:09:54.813+0800 INFO registrar/registrar.go:152 States Loaded from registrar: 0 2020-04-21T14:09:54.813+0800 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning. 2020-04-21T14:09:54.813+0800 INFO crawler/crawler.go:72 Loading Inputs: 1 2020-04-21T14:09:54.813+0800 INFO log/input.go:152 Configured paths: [/var/test.log] 2020-04-21T14:09:54.813+0800 INFO input/input.go:114 Starting input of type: log; ID: 13852086122507548810 2020-04-21T14:09:54.813+0800 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1 2020-04-21T14:09:54.813+0800 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
[root@redmine-dbs filebeat-7.6.2]# ps aux|grep filebeat-7.6.2 root 1422 0.0 1.4 260868 28608 pts/0 Sl 14:09 0:00 /root/filebeat-7.6.2/filebeat -c /root/filebeat-7.6.2/config.yml root 1482 0.0 0.0 103244 872 pts/0 S+ 14:11 0:00 grep filebeat-7.6.2 [root@redmine-dbs filebeat-7.6.2]# echo testlog > /var/test.log
安裝Kibana：
[root@localhost ~]# pwd /root [root@localhost ~]# tar -zxvf kibana-7.6.2-linux-x86_64.tar.gz [root@localhost ~]# mv kibana-7.6.2-linux-x86_64 kibana-7.6.2 [root@localhost ~]# cd kibana-7.6.2 [root@localhost kibana-7.6.2]# cp config/kibana.yml config/kibana.yml_bak [root@localhost kibana-7.6.2]# vi config/kibana.yml
kibana.yml只開啟下面幾個配置項
server.port: 5601 server.host: "172.16.1.81" elasticsearch.hosts: ["http://172.16.1.169:9200"] kibana.index: ".kibana"
啟動：
[root@localhost kibana-7.6.2]# ./bin/kibana --allow-root
瀏覽器訪問：http://192.16.1.81:5601
此時，到172.16.1.31 vi /var/test.log 給這個日誌檔案隨便寫新增幾行，在上面就看到日誌記錄，並且可以檢索
至此，ALK環境搭建完畢
&n