elasticsearch的版本要注意相容,以清華開源網站為例,不同版本的es有對應適應版本的filebeat、kibana等,這裡以8.50為例

 我這裡用filebeat直接代替logstash,架構是需要收集日誌的機器上安裝filebeat,多臺es機器組成叢集,數量最好不要低於3臺\

es:

官方文件連結:https://www.elastic.co/guide/en/elasticsearch/reference/8.5/settings.html

wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic-8.x/8.5.0/elasticsearch-8.5.0-x86_64.rpm
 

cd /etc/elasticsearch/ && cp elasticsearch.yml{,.bak} && vi elasticsearch.yml

#叢集 多臺是新增此配置
cluster.name: my-application

node.name: node-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 172.22.16.17,127.0.0.1

#http.port: 9200
#新增es主機,可以是ip:埠,或者域名
discovery.seed_hosts: [
 
"172.1.1.1", "172.1.1.2", "172.1.1.3"]
#指定master節點列表,當1掛掉提升2
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]

xpack.security.enabled: false
#預設
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: 
 
true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

http.host: 0.0.0.0

需要收集日誌的機器安裝Filebeat:

官方文件連結:https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic-8.x/8.5.0/filebeat-8.5.0-x86_64.rpm

修改配置檔案:

cd /etc/filebeat && cp filebeat.yml{,.bak}&& vi filebeat.yml

filebeat.inputs:
#7.16版本之後已經啟用log,現改filestream,同一機器下多個日誌ID不能相同,建立一個tags,後邊有tags區分要寫入不同的索引當中
- type: filestream
  id: my-online-id
  enabled: true
  paths:
    - /var/log/a.log
  tags: ["log-a"]

- type: filestream
  id: my-flyshenhe-id
  enabled: true
  paths:
    - /var/log/b.log
  tags: ["log-b"]

setup.ilm.enabled: false
setup.template.name: "log"
setup.template.pattern: "log-*"
#可設定分片和副本數,這是預設的
setup.template.settings:
  index.number_of_shards: 1

#輸出到es叢集,hosts填入主機ip
output.elasticsearch:
  hosts: ["172.1.1.1:9200", "172.1.1.2:9200", "172.1.1.3:9200"]
  indices:
    - index: "log-a-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "log-a"
    - index: "log-b-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "log-b"

展示到kibana:

cd /etc/kibana/ && cp kibana.yml{,.bak} && vi kibana.yml