Kafka+Zookeeper+Filebeat+ELK 搭建日誌收集系統

阿新 • • 發佈：2018-10-10

could not arch success div 名稱 fill pil ice oca

ELK

ELK目前主流的一種日誌系統,過多的就不多介紹了
Filebeat收集日誌,將收集的日誌輸出到kafka,避免網絡問題丟失信息
kafka接收到日誌消息後直接消費到Logstash
Logstash將從kafka中的日誌發往elasticsearch
Kibana對elasticsearch中的日誌數據進行展示

image

環境介紹:

軟件版本:
- Centos 7.4
- java 1.8.0_45
- Elasticsearch 6.4.0
- Logstash 6.4.0
- Filebeat 6.4.0
- Kibana 6.4.0
- Kafka 2.12
- Zookeeper 3.4.13

服務器:
- 10.241.0.1  squid(軟件分發,集中控制)
- 10.241.0.10 node1
- 10.241.0.11 node2
- 10.241.0.12 node3

部署角色
- elasticsearch: 10.241.0.10(master),10.241.0.11,10.241.0.12
  https://www.elastic.co/cn/products/elasticsearch
  Elasticsearch 允許執行和合並多種類型的搜索 （ 結構化、非結構化、地理位置、度量指標 ）搜索方式

- logstash: 10.241.0.10,10.241.0.11,10.241.0.12
  https://www.elastic.co/cn/products/logstash
  Logstash 支持各種輸入選擇 ，可以在同一時間從眾多常用來源捕捉事件

- filebeat: 10.241.0.10,10.241.0.11,10.241.0.12
  https://www.elastic.co/cn/products/beats/filebeat
  Filebeat 內置的多種模塊（auditd、Apache、NGINX、System 和 MySQL）可實現對常見日誌格式的一鍵收集、解析和可視化.

- kibana: 10.241.0.10
  https://www.elastic.co/cn/products/kibana
  Kibana 讓您能夠可視化 Elasticsearch 中的數據並操作 Elastic Stack

- kafka: 10.241.0.10,10.241.0.11,10.241.0.12
  http://kafka.apache.org/
  Kafka是一種高吞吐量的分布式發布訂閱消息系統，它可以處理消費者規模的網站中的所有動作流數據
  kafka集群部署前面的博客: https://www.jianshu.com/p/a9ff97dcfe4e

開始安裝部署ELK

1.下載安裝包及測試安裝包完整性

[root@squid ~]# cat /etc/hosts
10.241.0.1  squid
10.241.0.10 squid
10.241.0.11 node2
10.241.0.12 node3

[root@squid ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
[root@squid ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz.sha512
[root@squid ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
[root@squid ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz.sha512
[root@squid ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz
[root@squid ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz.sha512
[root@squid ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz
[root@squid ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz.sha512

[root@squid ~]# yum install perl-Digest-SHA
[root@squid ~]# shasum -a 512 -c  elasticsearch-6.4.0.tar.gz.sha512
elasticsearch-6.4.0.tar.gz: OK
[root@squid ~]# shasum -a 512 -c  filebeat-6.4.0-linux-x86_64.tar.gz.sha512
filebeat-6.4.0-linux-x86_64.tar.gz: OK
[root@squid ~]# shasum -a 512 -c  kibana-6.4.0-linux-x86_64.tar.gz.sha512
kibana-6.4.0-linux-x86_64.tar.gz: OK
[root@squid ~]# shasum -a 512 -c  logstash-6.4.0.tar.gz.sha512
logstash-6.4.0.tar.gz: OK

2.部署elasticsearch

1) Ansible主機清單
[root@squid ~]# cat /etc/ansible/hosts 
[client]
10.241.0.10 es_master=true
10.241.0.11 es_master=false
10.241.0.12 es_master=false

2) 創建es用戶和用戶組
[root@squid ~]# ansible client -m group -a ‘name=elk‘
[root@squid ~]# ansible client -m user -a ‘name=es group=elk home=/home/es shell=/bin/bash‘

3) 將elasticsearch解壓到目標主機
[root@squid ~]# ansible client -m unarchive -a ‘src=/root/elasticsearch-6.4.0.tar.gz  dest=/usr/local owner=es group=elk‘

4)將準備好的es配置文件模板分發到各個節點
[root@squid ~]# cat elasticsearch.yml.j2 
#集群名稱及數據存放位置
cluster.name: my_es_cluster
node.name: es-{{ansible_hostname}}
path.data: /data/elk/es/data
path.logs: /data/elk/es/logs
#允許跨域訪問
http.cors.enabled: true 
http.cors.allow-origin: "*" 
#集群中的角色
node.master: {{es_master}}
node.data: true 
#允許訪問的地址及傳輸使用的端口
network.host: 0.0.0.0
transport.tcp.port: 9300
#使用tcp傳輸壓縮
transport.tcp.compress: true
http.port: 9200
#使用單播模式去連接其他節點
discovery.zen.ping.unicast.hosts: ["node1","node2","node3"]

5) 執行ansible,分發配置文件
[root@squid ~]# ansible client -m template -a ‘src=/root/elasticsearch.yml.j2 dest=/usr/local/elasticsearch-6.4.0/config/elasticsearch.yml owner=es group=elk‘

6) 修改系統允許最大打開的文件句柄數等參數,
[root@squid ~]# cat change_system_args.sh
#!/bin/bash
if [ "`grep 65536 /etc/security/limits.conf`" = "" ]
then
cat >> /etc/security/limits.conf << EOF
# End of file
* - nofile 1800000
        * soft nproc 65536
        * hard nproc 65536
        * soft nofile 65536
        * hard nofile 65536
EOF
fi

if [ "`grep 655360 /etc/sysctl.conf`" = "" ]
then
echo "vm.max_map_count=655360"  >> /etc/sysctl.conf
fi

7) 通過ansible來執行腳本
[root@squid ~]# ansible client -m script -a ‘/root/change_system_args.sh‘

8) 重啟目標主機,是參數生效(因為目標主機重啟 所以ansible連不上)
[root@squid ~]# ansible client -m shell -a ‘reboot‘
10.241.0.11 | UNREACHABLE! => {
    "changed": false, 
    "msg": "SSH Error: data could not be sent to remote host \"10.241.0.11\". Make sure this host can be reached over ssh", 
    "unreachable": true
}
10.241.0.12 | UNREACHABLE! => {
    "changed": false, 
    "msg": "SSH Error: data could not be sent to remote host \"10.241.0.12\". Make sure this host can be reached over ssh",
    "unreachable": true
}
10.241.0.10 | UNREACHABLE! => {
    "changed": false, 
    "msg": "SSH Error: data could not be sent to remote host \"10.241.0.10\". Make sure this host can be reached over ssh",
    "unreachable": true
}

9 )創建elk目錄
[root@squid ~]# ansible client -m file -a ‘name=/data/elk/  state=directory owner=es group=elk‘

10) 啟動es
[root@squid ~]# ansible client -m shell -a ‘su - es -c "/usr/local/elasticsearch-6.4.0/bin/elasticsearch -d"‘ 

10.241.0.11 | SUCCESS | rc=0 >>

10.241.0.10 | SUCCESS | rc=0 >>

10.241.0.12 | SUCCESS | rc=0 >>

11) 查看是否啟動
[root@squid ~]# ansible client -m shell -a ‘ps -ef|grep elasticsearch‘ 
10.241.0.12 | SUCCESS | rc=0 >>
es        3553     1 19 20:35 ?        00:00:48 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.eFvx2dMC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch-6.4.0 -Des.path.conf=/usr/local/elasticsearch-6.4.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /usr/local/elasticsearch-6.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es        3594  3553  0 20:35 ?        00:00:00 /usr/local/elasticsearch-6.4.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root      3711  3710  0 20:39 ?        00:00:00 /bin/sh -c ps -ef|grep elasticsearch
root      3713  3711  0 20:39 ?        00:00:00 grep elasticsearch

10.241.0.10 | SUCCESS | rc=0 >>
es        4899     1 22 20:35 ?        00:00:54 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.1uRdvBGd -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch-6.4.0 -Des.path.conf=/usr/local/elasticsearch-6.4.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /usr/local/elasticsearch-6.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es        4940  4899  0 20:35 ?        00:00:00 /usr/local/elasticsearch-6.4.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root      5070  5069  0 20:39 ?        00:00:00 /bin/sh -c ps -ef|grep elasticsearch
root      5072  5070  0 20:39 ?        00:00:00 grep elasticsearch

10.241.0.11 | SUCCESS | rc=0 >>
es        3556     1 19 20:35 ?        00:00:47 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.fnAavDi0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch-6.4.0 -Des.path.conf=/usr/local/elasticsearch-6.4.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /usr/local/elasticsearch-6.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es        3597  3556  0 20:35 ?        00:00:00 /usr/local/elasticsearch-6.4.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root      3710  3709  0 20:39 ?        00:00:00 /bin/sh -c ps -ef|grep elasticsearch
root      3712  3710  0 20:39 ?        00:00:00 grep elasticsearch

12) 查看集群狀態
[root@squid ~]# curl -s http://node1:9200/_nodes/process?pretty |grep -C 5 _nodes
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "my_es_cluster",

3.部署Filebeat

1) 分發安裝包到客戶機
[root@squid ~]# ansible client -m unarchive -a ‘src=/root/filebeat-6.4.0-linux-x86_64.tar.gz dest=/usr/local‘

2) 修改安裝包名稱
[root@squid ~]# ansible client -m shell -a ‘mv /usr/local/filebeat-6.4.0-linux-x86_64 /usr/local/filebeat-6.4.0‘
10.241.0.12 | SUCCESS | rc=0 >>

10.241.0.11 | SUCCESS | rc=0 >>

10.241.0.10 | SUCCESS | rc=0 >>

3) 修改配置文件
[root@squid ~]# cat filebeat.yml.j2 
filebeat.prospectors:
- type: log
  paths:
    - /var/log/supervisor/kafka

output.kafka:
  enabled: true
  hosts: ["10.241.0.10:9092","10.241.0.11:9092","10.241.0.12:9092"]
  topic: kafka_run_log

##參數解釋
enabled 表明這個模塊是啟動的
host  把filebeat的數據發送到那臺kafka上
topic 這個很重要,發送給kafka的topic,若topic不存在,則會自動創建此topic

4) 分發到客戶機,並將原來的配置文件備份
[root@squid ~]# ansible client -m copy -a ‘src=/root/filebeat.yml.j2 dest=/usr/local/filebeat-6.4.0/filebeat.yml backup=yes‘

5) 啟動filebeat
[root@squid ~]# ansible client -m shell -a ‘/usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml &‘
10.241.0.11 | SUCCESS | rc=0 >>

10.241.0.10 | SUCCESS | rc=0 >>

10.241.0.12 | SUCCESS | rc=0 >>

6) 查看filebeat進程
[root@squid ~]# ansible client -m shell -a ‘ps -ef|grep filebeat| grep -v grep‘
10.241.0.12 | SUCCESS | rc=0 >>
root      4890     1  0 22:50 ?        00:00:00 /usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml

10.241.0.10 | SUCCESS | rc=0 >>
root      6881     1  0 22:50 ?        00:00:00 /usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml

10.241.0.11 | SUCCESS | rc=0 >>
root      4939     1  0 22:50 ?        00:00:00 /usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml

7) 查看是否有topic創建成功
[root@node1 local]# /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper  10.241.0.10:2181
ConsumerTest
__consumer_offsets
kafka_run_log #filebeat創建的topic
topicTest

4.部署Logstash

1) 解壓安裝包值目標主機
[root@squid ~]# ansible client -m unarchive -a ‘src=/root/logstash-6.4.0.tar.gz dest=/usr/local owner=es group=elk‘

2) Logstash配置文件
[root@squid ~]# cat logstash-kafka.conf.j2
input {
    kafka {
        type => "kafka-logs"
        bootstrap_servers => "10.241.0.10:9092,10.241.0.11:9092,10.241.0.12:9092"
        group_id => "logstash"
        auto_offset_reset => "earliest"
        topics => "kafka_run_log"
        consumer_threads => 5
        decorate_events => true
        }
}

output {
    elasticsearch {
    index => ‘kafka-run-log-%{+YYYY.MM.dd}‘
    hosts => ["10.241.0.10:9200","10.241.0.11:9200","10.241.0.12:9200"]
}

3) 使用ansible推送logstash配置文件到目標主機
[root@squid ~]# ansible client -m copy -a ‘src=/root/logstash.conf.j2 dest=/usr/local/logstash-6.4.0/config/logstash.conf owner=es group=elk‘

4) 啟動Logstash
[root@squid ~]# ansible client -m shell -a ‘su - es -c "/usr/local/logstash-6.4.0/bin/logstash -f /usr/local/logstash-6.4.0/config/logstash.conf &"‘     

5)_查看Logstash進程
[root@squid ~]# ansible client -m shell -a ‘ps -ef|grep logstash|grep -v grep‘
10.241.0.11 | SUCCESS | rc=0 >>
es        6040     1 99 23:39 ?        00:02:11 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash-6.4.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash-6.4.0/config/logstash.conf

10.241.0.12 | SUCCESS | rc=0 >>
es        5970     1 99 23:39 ?        00:02:13 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash-6.4.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash-6.4.0/config/logstash.conf

10.241.0.10 | SUCCESS | rc=0 >>
es        9095     1 98 23:39 ?        00:02:10 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash-6.4.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash-6.4.0/config/logstash.conf

5.部署kibana

1) 將安裝包拷貝到node1節點
[root@squid ~]# scp kibana-6.4.0-linux-x86_64.tar.gz [email protected]:/root
kibana-6.4.0-linux-x86_64.tar.gz                 100%  179MB  59.7MB/s   00:03

2) 解壓kibana
[root@node1 ~]# tar  -zxf kibana-6.4.0-linux-x86_64.tar.gz  -C /usr/local
[root@node1 ~]# mv /usr/local/kibana-6.4.0-linux-x86_64/ /usr/local/kibana-6.4.0

3) 修改配置文件
[root@node1 ~]# cat /usr/local/kibana-6.4.0/config/kibana.yml
server.port: 5601
server.host: "10.241.0.10"
kibana.index: ".kibana

4) 啟動kibana (前臺啟動)
[root@node1 ~]# /usr/local/kibana-6.4.0/bin/kibana

5) 訪問的kibana
http://10.241.0.10:5601

6) 添加日誌
Management -> Kibana 列Index Patterns -> Index pattern

7) 發送消息到kafka-run-log  topic,查看是否能通過kibana展示

image

image

作者：baiyongjie
鏈接：https://www.jianshu.com/p/d072a55aa844
來源：簡書
簡書著作權歸作者所有，任何形式的轉載都請聯系作者獲得授權並註明出處。

Kafka+Zookeeper+Filebeat+ELK 搭建日誌收集系統

could not arch success div 名稱 fill pil ice oca ELK ELK目前主流的一種日誌系統,過多的就不多介紹了 Filebeat收集日誌,將收集的日誌輸出到kafka,避免網絡問題丟失信息 kafka接收到日誌消息後直接消費到Lo

docker搭建日誌收集系統EFK

EFK Elasticsearch是一個數據搜尋引擎和分散式NoSQL資料庫的組合，提過日誌的儲存和搜尋功能。Fluentd是一個訊息採集，轉化，轉發工具，目的是提供中心化的日誌服務。Kibana是一個帶有強大資料整理，分析的web前端，將資料以視覺化的方式呈現給使用者。注意： doc

Filebeat + Elasitcsearch + Kibana 日誌收集系統

Filebeat 5.6.3 下載地址：https://www.elastic.co/downloads/beats/filebeat 安裝在需要蒐集的日誌機器 1.配置filebeat，vim /opt/software/filebeat-5.6.3-linux-x8

基於ELK的日誌收集系統的心得

elasticsearch+logstash+kinana搭建的日誌收集系統 elasticsearch是基於倒排序查詢的查詢引擎，什麼叫倒排序？比如mysql建立的索引是正排序，對於規範化資料(比如表格，元資料)而言基本使用正排序索引，倒排序一般用於文字之類的查詢，典型

Linux搭建ELK日誌收集系統：FIlebeat+Redis+Logstash+Elasticse

uri 對數 exp 取數網速 docker useradd 通過演示 Centos7部署ELK日誌收集系統一、ELK概述： ELK是一組開源軟件的簡稱，其包括Elasticsearch、Logstash 和 Kibana。ELK最近幾年發展迅速，已經成為目前最流行的

filebeat + kafka + logstash + Elasticsearch + Kibana日誌收集系統搭建

一、介紹在日常運維工作中，對於系統和業務日誌的處理尤為重要。今天，在這裡分享一下自己部署的filebeat + kafka + ELK開源實時日誌分析平臺的記錄過程。 1、ELK介紹 &nbs

ELK+kafka構建日誌收集系統

背景：最近線上上了ELK，但是隻用了一臺Redis在中間作為訊息佇列，以減輕前端es叢集的壓力，Redis的叢集解決方案暫時沒有接觸過，並且Redis作為訊息佇列並不是它的強項；所以最近將Redis換成了專業的訊息資訊釋出訂閱系統Kafka, Kafka的更多介紹大家可以看這裡：傳

基於flume+kafka+storm日誌收集系統搭建

基於flume+kafka+storm日誌收集系統搭建 1. 環境 192.168.0.2 hadoop1 192.168.0.3 hadoop2 192.168.0.4 hadoop3 已經

Logback+ELK+SpringMVC搭建日誌收集服務器

HA spring points ice 接口 eve bsp exe 路徑 (轉) 1.ELK是什麽？ ELK是由Elasticsearch、Logstash、Kibana這3個軟件的縮寫。 Elasticsearch是一個分布式搜索分析引擎，穩定、可水平擴展、易於管理

elk+kafka+zookeeper+filebeat安裝

ast install chown 推薦 sysconf ini module echo 需要 ElasticSearch6.0 ElasticSearch6.0安裝 #依賴jdk8 rpm -ivh elasticsearch-6.0.0.rpm vim /e

2018年ElasticSearch6.2.2教程ELK搭建日誌采集分析系統（目錄）

1.2 安裝包接口語句參數配置介紹 ofo 計劃裝包章節一 2018年 ELK課程計劃和效果演示 1、課程安排和效果演示簡介：課程介紹和主要知識點說明，ES搜索接口演示，部署的ELK項目演示章節二 elasticSearch 6.2版本基礎講

2018年ElasticSearch6.2.2教程ELK搭建日誌采集分析系統（教程詳情）

domain 服務器日誌環境解決對比修改 tst 阿裏雲服務們的章節一 2018年 ELK課程計劃和效果演示1、課程安排和效果演示簡介：課程介紹和主要知識點說明，ES搜索接口演示，部署的ELK項目演示es: localhost:9200kibana http:

快速搭建ELK 叢集日誌收集工具Centos7 +Logstash +Elasticsearch+Kibana 環境

應用場景：伺服器叢集，如果程式報錯，不懂從哪一臺伺服器上檢視日誌；想更加直接了當的看到日誌，什麼伺服器，什麼錯誤。學到什麼通過這篇文章可以傻瓜式的安裝程式，並且瞭解分散式應用日誌收集的場景。環境嶄新的一臺centos7虛擬機器；介紹介紹就算了，搜得到這個的大概知道是幹嘛的

elk日誌收集系統，開機自動啟動

以Elasticsearch為例：建立一個開啟es指令碼 es.sh ES_HOME=/system/es/elasticsearch-6.4.2 #es存放的目錄 sudo -iu elsearch $ES_HOME/bin/elasticsearch 開機指令碼：

ELK日誌收集系統，看別人的，學習了

這裡先簡單介紹下什麼是ELK。 ELK（ElasticSearch, Logstash, Kibana），三者組合在一起搭建實時的日誌分析平臺。 1、 Elasticsearch 是個開源分散式搜尋引擎，它的特點有：分散式，零配置，自動發現，索引自動分片，索引副本機制，

elk搭建日誌系統

一、使用背景當生產環境有很多伺服器、很多業務模組的日誌需要每時每刻檢視時二、環境系統：centos 6.5 JDK：1.8 Elasticsearch-5.0.0 Logstash-5.0.0 k

搭建日誌分析系統ELK

日誌分析元件3件套： E：Elasticsearch是一個搜尋引擎，基於Lucene，天然分散式，很容易水平擴充套件，遮蔽了複雜的分散式概念，對外提供RESTfulAPI。 L：Logs

從零開始搭建一個ELKB日誌收集系統

當今的軟體開發多核以及分佈已經成為了常態，基本上稍大型的應用都是多臺機器分散式部署。分散式在提高效能的同時也帶來了很多問題，今天我們只討論一點，那就是如何處理多臺機器線上系統的日誌。以我司的某個應用T為例，部署在了百度雲5臺機子上，其中一臺擁有公網IP，使用了百度雲提供的負載均衡服務。每次想要在日誌中檢索

logstash+elasticsearch+kibana搭建日誌收集分析系統

input { file { type => "api_log" path => "/home/jws/app/nginxserver/logs/apiaccess.log" debug

nginx+flume+hdfs搭建實時日誌收集系統

1、配置nginx.conf，新增以下配置 http { #配置日誌格式 log_format lf '$remote_addr^A$msec^A$http_host^A$reques

Kafka+Zookeeper+Filebeat+ELK 搭建日誌收集系統

ELK

環境介紹:

開始安裝部署ELK

1.下載安裝包及測試安裝包完整性

2.部署elasticsearch

3.部署Filebeat

4.部署Logstash

5.部署kibana

相關推薦