• 環境
  
  • centos6.9
  • ELK5.6
  • 所有節點都是單點非叢集
    
    • filebeat：10.99.2.16
    • elk：10.99.2.17
    • kafka：10.99.2.23
  • 官方文件
• es安裝

yum安裝java環境和es：

yum install elasticsearch-5.6.7.rpm jre-8u161-linux-x64.rpm -y
yum安裝的需要普通使用者開啟
useradd elk
cp -r /etc/elasticsearch /usr/share/elasticsearch/config
chown -R elk:elk /usr/share/elasticsearch
chown -R
 
 elk:elk /etc/elasticsearch

配置檔案

1、在/etc/security/limits.conf中新增：
 3.           soft   nproc        655350
 4.           hard   nproc        655350
2、在/etc/security/limits.d/90-nproc.conf中修改：
 5.          soft    nproc     4096
root       soft    nproc     unlimited
3、修改es配置檔案jvm.options和elasticsearch.yml
# jvm.options：
 

    -Xms2g
    -Xmx2g
# elasticsearch.yml：
    network.host: 0.0.0.0
    http.port: 9200
    bootstrap.system_call_filter: false

啟動

su elk      切換使用者
/usr/share/elasticsearch/bin/elasticsearch 
加上"-d"是後臺執行，不加可以看到無法啟動的報錯資訊

• logstash安裝

yum安裝

 yum install -y logstash-5.6.7.rpm

配置檔案 /etc/logstash/logstash.yml

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d

配置檔案 /etc/logstash/conf.d/logstash_agent.conf

input {
  kafka {
    bootstrap_servers => "10.99.2.30:9092"
    topics => "service-pv"
 #   group_id => "logstash-lf"
    codec => "json"
  }
}
filter {
    json {
      source => "message"
      remove_field => "message"
    }
}
output {
    elasticsearch {
    hosts => "10.99.2.17:9200"
    }
    stdout { codec => rubydebug }
}

啟動

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_agent.conf 

kibana安裝
yum安裝

yum install kibana-5.6.5-x86_64.rpm

配置檔案 /etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"

啟動

/usr/share/kibana/bin/kibana

• kafka
  kafka下載地址
  也需要java環境，安裝：

tar -zxf  kafka_2.11-1.0.0.tgz
cd kafka_2.11-1.0.0
啟動zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
啟動kafka
bin/kafka-server-start.sh config/server.properties

1、建立topic 
$bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test 

2、檢視建立的topic 
$ bin/kafka-topics.sh -list -zookeeper localhost:2181 
test

3、生產訊息測試 
$bin/kafka-console-producer.sh --broker-list 192.168.12.105:9092 --topic test 
this is test          #輸入後回車

4、消費訊息測試 
$bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
this is test

下載安裝

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.7-x86_64.rpm
yum install -y filebeat-5.6.7-x86_64.rpm

修改配置檔案 /etc/filebeat/filebeat.yml

- input_type: log
  paths:
    - /usr/local/lnmp/tengine/logs/access_json.log
# 要採集的日誌檔案
output.kafka:
    hosts: ["10.99.2.30:9092"]
    enable: true
    topic: service-pv
# 輸出到kafka

測試配置檔案

filebeat.sh -configtest -e

啟動

/etc/init.d/filebeat start